Purpose of the DevOps Stack

Introduction

Since about 2006, the IT industry has been totally disrupted by the conjunction of several factors, including the virtualization of resources and the possibility of obtaining them on-demand. This new way of managing IT resources is widely known as "Cloud Computing". It has become a kind of standard, or at least an objective pursued also by people wishing to maintain on-premises infrastructures. The long-awaited dream of finally being able to consider IT as a simple commodity. Well that’s what many believed.

The concrete consequences of this revolution for IT companies like Camptocamp have mainly been the incredible growth in the number of resources to be provisioned and managed. Extensive process automation engaged by our teams allowed us to cope with such a growth without lowering our quality standards, therefore remaining competitive.

We had no respite in this automation journey and made the best out of a thriving ecosystem of open source projects that address all the needs related to this paradigm shift.

We have learned a lot during all these years and the DevOps Stack is our answer to implement our best practices in a containerized world where Kubernetes is the de facto standard for application platforms.

Cloud Agnostic Kubernetes Distribution

Nowadays, the question is not so much whether there is software to solve a problem, but which one to use. Like a Linux distribution, we have container orchestrators (Kubernetes as a kernel) and many other projects revolving around it and offering different features. These features are sometimes oriented towards developers, operators or both (e.g. observability tools).

As for a Linux distribution, what we need is consistency, components that work together and that are easy to install and maintain. But unlike a Linux distribution which is by definition server-centric, the promise of containers and Kubernetes is portability. The possibility of moving from one cloud (or on-prem infrastructure) to another, of distributing the workloads on different platforms. At the end, this DevOps Stack is simply a "Cloud Agnostic Kubernetes Distribution".

Without the DevOps Stack

Without the DevOps Stack

The illustration above tries to show that while it is quite simple to manage one Kubernetes project, the complexity increases dramatically when you add more and more projects and you want to keep a certain homogeneity between them. The main problems observed are:

  • It does not scale

  • There is a lot of code duplications

  • There is no implementation reference

  • Improvements are not always backported to other projects

  • It is very hard to maintain

  • All projects will eventually diverge

  • Each project points to different versions of dependencies

This is not a new IT problem. Two decades ago, it was not really easy to manage thousands, or even hundreds of hosts in a homogeneous way. This problem has been successfully addressed by solutions like CFEngine, Puppet or more recently by Ansible. Today, we must adapt these principles for Kubernetes. There are many open source components out there, we just a smart way to interconnect them.

With the DevOps Stack

With the DevOps Stack

By creating the DevOps Stack (a kind of abstraction layer), it becomes possible to instantiate all the components through a logical and coherent process. The main advantages are:

  • It scales

  • Minimal code duplication

  • A reference implementation

  • Improvements can be easily backported to other projects

  • Dependency pining is done on the DevOps Stack to ensure reproducibility

Summary

In the end, the DevOps Stack provides:

  • A Playground to test/prototype new stuff related to k8s world

  • An opinionated abstraction layer to deploy and manage our Kubernetes deployments

  • Enforce DevOps best practices:

    • Infrastructure as Code (IaC)

    • Configuration Management (CM)

    • GitOps

    • Continuous Integration (CI)

    • Continuous Delivery (CD)

    • Documentation as Code

  • A consistent toolset

    • Observability (monitoring, logging, alerting, trending)

    • Continuous Delivery (change management + Git features)

    • Dynamic Web Routing

    • Dynamic Storage Provisioning

    • Dynamic Certificate Management

    • Authentication and Authorization

With the following main features:

  • Supports multiple Kubernetes distributions (AWS EKS, Azure AKS, Exoscale SKS and KinD for locals development and testing purposes);

  • Supports blue/green or prod/qa/int/dev patterns;

  • Infrastructure as Code with Terraform;

  • Automatic deployment of "core apps" (cert-manager, Prometheus, AlertManager, Grafana, Loki, Traefik. etc.) in GitOps spirit with Argo CD;

  • Central point of documentation deployed with Antora.