SKS Variant

This folder contains the variant to use when deploying in Exoscale using an SKS cluster.

Usage

This module can be declared by adding the following block on your Terraform configuration:

module "kube-prometheus-stack" {
  source = "git::https://github.com/camptocamp/devops-stack-module-kube-prometheus-stack//sks?ref=<RELEASE>"

  cluster_name     = module.sks.cluster_name
  base_domain      = module.sks.base_domain
  cluster_issuer   = local.cluster_issuer
  argocd_namespace = module.argocd_bootstrap.argocd_namespace

  prometheus = {
    oidc = module.oidc.oidc
  }
  alertmanager = {
    oidc = module.oidc.oidc
  }
  grafana = {
    oidc = module.oidc.oidc
  }

  dependency_ids = {
    argocd       = module.argocd_bootstrap.id
    traefik      = module.traefik.id
    cert-manager = module.cert-manager.id
    keycloak     = module.keycloak.id
    oidc         = module.oidc.id
    longhorn     = module.longhorn.id
    loki-stack   = module.loki-stack.id
  }
}

When also deploying Thanos in the same cluster, you need to configure the metrics_storage variable with the values of the bucket created for the Thanos module. This will automatically activate the Thanos sidecar in the Prometheus pods as well as defining Thanos as the default data source for Grafana.

module "kube-prometheus-stack" {
  source = "git::https://github.com/camptocamp/devops-stack-module-kube-prometheus-stack//sks?ref=<RELEASE>"

  cluster_name     = module.sks.cluster_name
  base_domain      = module.sks.base_domain
  cluster_issuer   = local.cluster_issuer
  argocd_namespace = module.argocd_bootstrap.argocd_namespace

  metrics_storage = {
    bucket_name = resource.aws_s3_bucket.this["thanos"].id
    region      = resource.aws_s3_bucket.this["thanos"].region
    access_key  = resource.exoscale_iam_access_key.s3_iam_key["thanos"].key
    secret_key  = resource.exoscale_iam_access_key.s3_iam_key["thanos"].secret
  }

  prometheus = {
    oidc = module.oidc.oidc
  }
  alertmanager = {
    oidc = module.oidc.oidc
  }
  grafana = {
    oidc = module.oidc.oidc
  }

  dependency_ids = {
    argocd       = module.argocd_bootstrap.id
    traefik      = module.traefik.id
    cert-manager = module.cert-manager.id
    keycloak     = module.keycloak.id
    oidc         = module.oidc.id
    longhorn     = module.longhorn.id
    loki-stack   = module.loki-stack.id
  }
}
Check the SKS deployment example to see how to create the S3 bucket and to better understand the values passed on the example above.

OIDC

This module was developed with OIDC in mind.

There is an OIDC proxy container deployed as a sidecar on the pods of Prometheus and Alertmanager. As such, the prometheus and alertmanager variables are expected to have a map oidc containing at least the Issuer URL, the Client ID, and the Client Secret.

As for Grafana, the OIDC configuration is done through the grafana variable. The oidc map is expected to contain the same values as for Prometheus and Alertmanager, but also the oauth_url, token_url and api_url values.

You can pass these values by pointing an output from another module (as above), or by defining them explicitly:

module "kube-prometheus-stack" {
  ...
  prometheus | alertmanager = {
    oidc = {
      issuer_url    = "<URL>"
      client_id     = "<ID>"
      client_secret = "<SECRET>"
    }
  }
  grafana = {
    oidc = {
      issuer_url    = "<URL>"
      client_id     = "<ID>"
      client_secret = "<SECRET>"
      oauth_url     = "<URL>"
      token_url     = "<URL>"
      api_url       = "<URL>"
    }
  }
  ...
}

Technical Reference

Dependencies

module.argocd_bootstrap.id

Obviously, the module depends on an already running Argo CD in the cluster in order for the application to be created.

module.traefik.id and module.cert-manager.id

This module has multiple ingresses and consequently it must be deployed after the module traefik and cert-manager.

module.keycloak.id and module.oidc.id

When using Keycloak as an OIDC provider for the Longhorn Dashboard, you need to add Keycloak and the OIDC module as dependencies.

module.longhorn.id

This module requires a Persistent Volume so it needs to be deployed after the module Longhorn.

module.loki-stack.id

In order to be able to check the logs collected by Loki in the Grafana interface, this module requires to be deployed after the module Loki, so it can detect it as a data source.

Requirements

The following requirements are needed by this module:

Modules

The following Modules are called:

kube-prometheus-stack

Source: ../

Version:

Required Inputs

The following input variables are required:

cluster_name

Description: Name given to the cluster. Value used for naming some the resources created by the module.

Type: string

base_domain

Description: Base domain of the cluster. Value used for the ingress' URL of the application.

Type: string

Optional Inputs

The following input variables are optional (have default values):

metrics_storage

Description: Exoscale SOS bucket configuration values for the bucket where the archived metrics will be stored.

Type:

object({
    bucket_name = string
    region      = string
    access_key  = string
    secret_key  = string
  })

Default: null

subdomain

Description: Subdomain of the cluster. Value used for the ingress' URL of the application.

Type: string

Default: "apps"

argocd_project

Description: Name of the Argo CD AppProject where the Application should be created. If not set, the Application will be created in a new AppProject only for this Application.

Type: string

Default: null

argocd_labels

Description: Labels to attach to the Argo CD Application resource.

Type: map(string)

Default: {}

destination_cluster

Description: Destination cluster where the application should be deployed.

Type: string

Default: "in-cluster"

target_revision

Description: Override of target revision of the application chart.

Type: string

Default: "v11.0.0"

cluster_issuer

Description: SSL certificate issuer to use. Usually you would configure this value as letsencrypt-staging or letsencrypt-prod on your root *.tf files.

Type: string

Default: "selfsigned-issuer"

helm_values

Description: Helm chart value overrides. They should be passed as a list of HCL structures.

Type: any

Default: []

deep_merge_append_list

Description: A boolean flag to enable/disable appending lists instead of overwriting them.

Type: bool

Default: false

app_autosync

Description: Automated sync options for the Argo CD Application resource.

Type:

object({
    allow_empty = optional(bool)
    prune       = optional(bool)
    self_heal   = optional(bool)
  })

Default:

{
  "allow_empty": false,
  "prune": true,
  "self_heal": true
}

dependency_ids

Description: n/a

Type: map(string)

Default: {}

resources

Description: Resource limits and requests for kube-prometheus-stack’s components. Follow the style on official documentation to understand the format of the values.

These are not production values. You should always adjust them to your needs.

Type:

object({

    prometheus = optional(object({
      requests = optional(object({
        cpu    = optional(string, "250m")
        memory = optional(string, "512Mi")
      }), {})
      limits = optional(object({
        cpu    = optional(string)
        memory = optional(string, "1024Mi")
      }), {})
    }), {})

    prometheus_operator = optional(object({
      requests = optional(object({
        cpu    = optional(string, "50m")
        memory = optional(string, "128Mi")
      }), {})
      limits = optional(object({
        cpu    = optional(string)
        memory = optional(string, "128Mi")
      }), {})
    }), {})

    thanos_sidecar = optional(object({
      requests = optional(object({
        cpu    = optional(string, "100m")
        memory = optional(string, "256Mi")
      }), {})
      limits = optional(object({
        cpu    = optional(string)
        memory = optional(string, "512Mi")
      }), {})
    }), {})

    alertmanager = optional(object({
      requests = optional(object({
        cpu    = optional(string, "50m")
        memory = optional(string, "128Mi")
      }), {})
      limits = optional(object({
        cpu    = optional(string)
        memory = optional(string, "256Mi")
      }), {})
    }), {})

    kube_state_metrics = optional(object({
      requests = optional(object({
        cpu    = optional(string, "50m")
        memory = optional(string, "128Mi")
      }), {})
      limits = optional(object({
        cpu    = optional(string)
        memory = optional(string, "128Mi")
      }), {})
    }), {})

    grafana = optional(object({
      requests = optional(object({
        cpu    = optional(string, "250m")
        memory = optional(string, "512Mi")
      }), {})
      limits = optional(object({
        cpu    = optional(string)
        memory = optional(string, "512Mi")
      }), {})
    }), {})

    node_exporter = optional(object({
      requests = optional(object({
        cpu    = optional(string, "50m")
        memory = optional(string, "128Mi")
      }), {})
      limits = optional(object({
        cpu    = optional(string)
        memory = optional(string, "128Mi")
      }), {})
    }), {})

  })

Default: {}

grafana

Description: Grafana settings

Type: any

Default: {}

prometheus

Description: Prometheus settings

Type: any

Default: {}

alertmanager

Description: Object containing Alertmanager settings. The following attributes are supported:

  • enabled: whether Alertmanager is deployed or not (default: true).

  • domain: domain name configured in the Ingress (default: prometheus.apps.${var.cluster_name}.${var.base_domain}).

  • oidc: OIDC configuration to be used by OAuth2 Proxy in front of Alertmanager (required).

  • deadmanssnitch_url: url of a Dead Man’s Snitch service Alertmanager should report to (by default this reporing is disabled).

  • slack_routes: list of objects configuring routing of alerts to Slack channels, with the following attributes:

  • name: name of the configured route.

  • channel: channel where the alerts will be sent (with '#').

  • api_url: slack URL you received when configuring a webhook integration.

  • matchers: list of strings for filtering which alerts will be sent.

  • continue: whether an alert should continue matching subsequent sibling nodes.

Type: any

Default: {}

metrics_storage_main

Description: Storage settings for the Thanos sidecar. Needs to be of type any because the structure is different depending on the variant used.

Type: any

Default: {}

dataproxy_timeout

Description: Variable to set the time when a query times out. This applies to all the Grafana’s data sources and can be manually configured per data source if desired.

Type: number

Default: 30

Outputs

The following outputs are exported:

id

Description: ID to pass other modules in order to refer to this module as a dependency.

grafana_admin_password

Description: The admin password for Grafana.

Reference in table format

Show tables

= Requirements

Name Version

argocd

>= 5

kubernetes

>= 2

null

>= 3

random

>= 3

utils

>= 1

= Modules

Name Source Version

kube-prometheus-stack

= Inputs

Name Description Type Default Required

metrics_storage

Exoscale SOS bucket configuration values for the bucket where the archived metrics will be stored.

 
object({
    bucket_name = string
    region      = string
    access_key  = string
    secret_key  = string
  })

null

no

cluster_name

Name given to the cluster. Value used for naming some the resources created by the module.

string

n/a

yes

base_domain

Base domain of the cluster. Value used for the ingress' URL of the application.

string

n/a

yes

subdomain

Subdomain of the cluster. Value used for the ingress' URL of the application.

string

"apps"

no

argocd_project

Name of the Argo CD AppProject where the Application should be created. If not set, the Application will be created in a new AppProject only for this Application.

string

null

no

argocd_labels

Labels to attach to the Argo CD Application resource.

map(string)

{}

no

destination_cluster

Destination cluster where the application should be deployed.

string

"in-cluster"

no

target_revision

Override of target revision of the application chart.

string

"v11.0.0"

no

cluster_issuer

SSL certificate issuer to use. Usually you would configure this value as letsencrypt-staging or letsencrypt-prod on your root *.tf files.

string

"selfsigned-issuer"

no

helm_values

Helm chart value overrides. They should be passed as a list of HCL structures.

any

[]

no

deep_merge_append_list

A boolean flag to enable/disable appending lists instead of overwriting them.

bool

false

no

app_autosync

Automated sync options for the Argo CD Application resource.

 
object({
    allow_empty = optional(bool)
    prune       = optional(bool)
    self_heal   = optional(bool)
  })
 
{
  "allow_empty": false,
  "prune": true,
  "self_heal": true
}

no

dependency_ids

n/a

map(string)

{}

no

resources

Resource limits and requests for kube-prometheus-stack’s components. Follow the style on official documentation to understand the format of the values.

These are not production values. You should always adjust them to your needs. 
object({

    prometheus = optional(object({
      requests = optional(object({
        cpu    = optional(string, "250m")
        memory = optional(string, "512Mi")
      }), {})
      limits = optional(object({
        cpu    = optional(string)
        memory = optional(string, "1024Mi")
      }), {})
    }), {})

    prometheus_operator = optional(object({
      requests = optional(object({
        cpu    = optional(string, "50m")
        memory = optional(string, "128Mi")
      }), {})
      limits = optional(object({
        cpu    = optional(string)
        memory = optional(string, "128Mi")
      }), {})
    }), {})

    thanos_sidecar = optional(object({
      requests = optional(object({
        cpu    = optional(string, "100m")
        memory = optional(string, "256Mi")
      }), {})
      limits = optional(object({
        cpu    = optional(string)
        memory = optional(string, "512Mi")
      }), {})
    }), {})

    alertmanager = optional(object({
      requests = optional(object({
        cpu    = optional(string, "50m")
        memory = optional(string, "128Mi")
      }), {})
      limits = optional(object({
        cpu    = optional(string)
        memory = optional(string, "256Mi")
      }), {})
    }), {})

    kube_state_metrics = optional(object({
      requests = optional(object({
        cpu    = optional(string, "50m")
        memory = optional(string, "128Mi")
      }), {})
      limits = optional(object({
        cpu    = optional(string)
        memory = optional(string, "128Mi")
      }), {})
    }), {})

    grafana = optional(object({
      requests = optional(object({
        cpu    = optional(string, "250m")
        memory = optional(string, "512Mi")
      }), {})
      limits = optional(object({
        cpu    = optional(string)
        memory = optional(string, "512Mi")
      }), {})
    }), {})

    node_exporter = optional(object({
      requests = optional(object({
        cpu    = optional(string, "50m")
        memory = optional(string, "128Mi")
      }), {})
      limits = optional(object({
        cpu    = optional(string)
        memory = optional(string, "128Mi")
      }), {})
    }), {})

  })

{}

no

grafana

Grafana settings

any

{}

no

prometheus

Prometheus settings

any

{}

no

alertmanager

Object containing Alertmanager settings. The following attributes are supported:

  • enabled: whether Alertmanager is deployed or not (default: true).

  • domain: domain name configured in the Ingress (default: prometheus.apps.${var.cluster_name}.${var.base_domain}).

  • oidc: OIDC configuration to be used by OAuth2 Proxy in front of Alertmanager (required).

  • deadmanssnitch_url: url of a Dead Man’s Snitch service Alertmanager should report to (by default this reporing is disabled).

  • slack_routes: list of objects configuring routing of alerts to Slack channels, with the following attributes:

  • name: name of the configured route.

  • channel: channel where the alerts will be sent (with '#').

  • api_url: slack URL you received when configuring a webhook integration.

  • matchers: list of strings for filtering which alerts will be sent.

  • continue: whether an alert should continue matching subsequent sibling nodes.

any

{}

no

metrics_storage_main

Storage settings for the Thanos sidecar. Needs to be of type any because the structure is different depending on the variant used.

any

{}

no

dataproxy_timeout

Variable to set the time when a query times out. This applies to all the Grafana’s data sources and can be manually configured per data source if desired.

number

30

no

= Outputs

Name Description

id

ID to pass other modules in order to refer to this module as a dependency.

grafana_admin_password

The admin password for Grafana.