devops-stack-module-cluster-sks

A DevOps Stack module to deploy a Kubernetes cluster on Exoscale SKS.

The module creates a Kubernetes cluster with the node pools passed as input. It also creates anti-affinity group for each node pool, a Network Load Balancer (NLB), and a security group for the entire cluster.

Usage

This module can be used with the following declaration:

module "sks" {
  source = "git::https://github.com/camptocamp/devops-stack-module-cluster-sks.git?ref=<RELEASE>"

  cluster_name       = local.cluster_name
  kubernetes_version = local.kubernetes_version
  zone               = local.zone
  base_domain        = local.base_domain

  nodepools = {
    "${local.cluster_name}-default" = {
      size            = 3
      instance_type   = "standard.large"
      description     = "Default node pool for ${local.cluster_name}."
      instance_prefix = "default"
    },
  }
}
A minimum of a single node pool with 3 nodes is required. See the Persistent Volumes section for more information.

Multiple node pools and with more complex settings can be declared. The following example adds a node pool with a taint and label to be used exclusively for monitoring workloads:

module "sks" {
  source = "git::https://github.com/camptocamp/devops-stack-module-cluster-sks.git?ref=<RELEASE>"

  cluster_name       = local.cluster_name
  kubernetes_version = local.kubernetes_version
  zone               = local.zone
  base_domain        = local.base_domain

  nodepools = {
    "${local.cluster_name}-default" = {
      size            = 3
      instance_type   = "standard.large"
      description     = "Default node pool for ${local.cluster_name}."
      instance_prefix = "default"
    },
    "${local.cluster_name}-monitoring" = {
      size            = 2
      instance_type   = "standard.large"
      description     = "Monitoring node pool for ${local.cluster_name}."
      instance_prefix = "monitoring"
      disk_size       = 150
      labels = {
        role = "monitoring"
      }
      taints = {
        nodepool = "monitoring:NoSchedule"
      }
    },
  }
}
You can consult the available instance types on the Pricing page of Exoscale. Note that not all instance types are available in all zones and take note of the these limitations of SKS.

Kubeconfig

The module uses the exoscale_sks_kubeconfig resource to get a Kubeconfig with administrator privileges on the cluster. We use this configuration file to parse the credentials needed to set the providers on the caller module.

If the variable create_kubeconfig_file is activated, a local file is created with the Kubeconfig content, which can be used to interact with the cluster. You can still get the Kubeconfig’s important values from the outputs kubernetes_*.
The validity of the client certificate is limited to 30 days by default and the earliest time for renewal is 10 days. This means that if no terraform apply is run between the 10th and 30th days after the last terraform apply, the Kubeconfig will be invalid and the next on will fail. There are variables available to customize these values to suit your needs.

In the case that you have left your Kubeconfig pass the expiry time, you can still get a new one by running a targeted terraform apply on the module:

$ terraform apply -target module.sks.exoscale_sks_kubeconfig.this

DNS and the base_domain variable

In production, this module requires a domain name to be passed in the base_domain variable. This module will take care to create a CNAME record that points to the NLB, using the cluster name as an hostname.

The DNS zone should be created outside this module and it requires a DNS Subscription in the same Exoscale account. This can be added on the Exoscale portal, on the DNS tab. The subscription needs to be manually activated on the web interface, but it is recommended that the DNS zone is created on your root Terraform module.

Check the Terraform code of the SKS example to learn how to create the DNS zone.

If no value is passed to the base_domain variable, the module will create a nip.io domain prefixed with the IP of the NLB. You can check said domain in the base_domain output.

Network Load Balancer and ingress traffic

The NLB is created in this module without any service associated. The NLB services are created by the Exoscale Cloud Controller Manager (CCM), which is deployed by default on SKS clusters. The CCM takes into account annotations on the LoadBalancer services to create the corresponding services on the NLB. These annotations are added by the Traefik module and for that reason you need to pass the outputs nlb_id, router_nodepool_id and router_instance_pool_id from this module to the Traefik module.

Check the official documentation of the CCM and this blog post to learn more. An example is also provided by Exoscale on the SKS documentation, which contains the required annotations as well was a few interesting comments.

Persistent Volumes

This module requires that you deploy Longhorn in order to have a way to provision persistent volumes for your workloads. We configured the Longhorn module to replicate the volumes at least 3 times throughout the available nodes. For that reason, you need to deploy at least a node pool with minimum 3 nodes.

Upgrading the cluster

The official documentation is a good starting point to understand the upgrade process of SKS clusters.

Manual upgrade of a minor Kubernetes version

  1. On your root Terraform code change the Kubernetes version deployed by your SKS module and do a terraform apply. This will upgrade the version of the control plane of the SKS cluster.

  2. Scale up all your node pools (router one included) through the size parameter on the nodepools and router_nodepool variables to twice their original size and do a terraform apply.

  3. Wait for all new nodes to be in a ready state and check that their Kubernetes version match the one you configured. Check in Longhorn Dashboard that all the nodes are schedulable. It is advised you to do a backup of all your volumes in case of troubles during the upgrade to avoid losing your applications persistent volumes.

  4. In the Longhorn dashboard, go to the Volume tab, select all your volumes and select Update Replicas Count action. In the dialog box, replace the actual replicas count of these volumes by twice your old schedulable node count (by default it’s 3) in order to replicate your volumes on the new nodes.

  5. Cordon all the old nodes and start draining them one by one using kubectl drain --ignore-daemonsets --delete-emptydir-data --timeout=1m <node_id>. This will move all the pods to the new nodes.

  6. When all the old nodes are drained and all pods are deployed to new nodes, do a terraform refresh. If you use a Keycloak module provisioned by Terraform with Keycloak provider you should have diffs on Keycloak’s resources. Apply them.

  7. Before deleting the old nodes, be sure to test and validate your cluster health! Once you’re confident enough, you can restore original node pool sizes in Terraform and apply. This will delete the old nodes.

  8. Finally, go to the Longhorn dashboard, restore the original replicas count for every volumes and check that every volumes are in healthy state.

SKS instance pools will automatically choose cordoned nodes to delete in priority.

Technical Reference

Requirements

The following requirements are needed by this module:

Providers

The following providers are used by this module:

Resources

The following resources are used by this module:

Required Inputs

The following input variables are required:

cluster_name

Description: The name of the Kubernetes cluster to create.

Type: string

zone

Description: The name of the zone where to deploy the SKS cluster. Available zones can be consulted here.

Type: string

kubernetes_version

Description: Kubernetes version to use for the SKS cluster. See exo compute sks versions for reference. May only be set at creation time.

Type: string

Optional Inputs

The following input variables are optional (have default values):

base_domain

Description: The base domain used for Ingresses. If not provided, nip.io will be used taking the NLB IP address.

Type: string

Default: null

description

Description: A free-form string description to apply to the SKS cluster.

Type: string

Default: null

auto_upgrade

Description: Enable automatic upgrade of the SKS cluster control plane.

Type: bool

Default: false

service_level

Description: Choose the service level for the SKS cluster. Starter can be used for test and development purposes, Pro is recommended for production workloads. The official documentation is available here.

Type: string

Default: "pro"

nodepools

Description: Map containing the SKS node pools to create.

Needs to be a map of maps, where the key is the name of the node pool and the value is a map containing at least the keys instance_type and size. The other keys are optional: description, instance_prefix, disk_size, labels, taints and private_network_ids. Check the official documentation here for more information.

Type:

map(object({
    size                = number
    instance_type       = string
    description         = optional(string)
    instance_prefix     = optional(string, "pool")
    disk_size           = optional(number, 50)
    labels              = optional(map(string), {})
    taints              = optional(map(string), {})
    private_network_ids = optional(list(string), [])
  }))

Default: null

router_nodepool

Description: Configuration of the router node pool. The defaults of this variable are sensible and rarely need to be changed. The variable is mainly used to change the size of the node pool when doing cluster upgrades.

Type:

object({
    size            = number
    instance_type   = string
    instance_prefix = optional(string, "router")
    disk_size       = optional(number, 20)
    labels          = optional(map(string), {})
    taints = optional(map(string), {
      nodepool = "router:NoSchedule"
    })
    private_network_ids = optional(list(string), [])
  })

Default:

{
  "instance_type": "standard.small",
  "size": 2
}

tcp_node_ports_world_accessible

Description: Create a security group rule that allows world access to to NodePort TCP services. Recommended to leave open as per SKS documentation.

Type: bool

Default: true

udp_node_ports_world_accessible

Description: Create a security group rule that allows world access to to NodePort UDP services.

Type: bool

Default: false

cni

Description: Specify which CNI plugin to use (cannot be changed after the first deployment). Accepted values are calico or cilium. This module creates the required security group rules.

Type: string

Default: "cilium"

kubeconfig_ttl

Description: Validity period of the Kubeconfig file in seconds. See official documentation for more information.

Type: number

Default: 0

kubeconfig_early_renewal

Description: Renew the Kubeconfig file if its age is older than this value in seconds. See official documentation for more information.

Type: number

Default: 0

create_kubeconfig_file

Description: Create a Kubeconfig file in the directory where terraform apply is run. The file will be named <cluster_name>-config.yaml.

Type: bool

Default: false

Outputs

The following outputs are exported:

cluster_name

Description: Name of the SKS cluster.

base_domain

Description: The base domain for the SKS cluster.

cluster_id

Description: ID of the SKS cluster.

nlb_ip_address

Description: IP address of the Network Load Balancer.

nlb_id

Description: ID of the Network Load Balancer.

router_nodepool_id

Description: ID of the node pool specifically created for Traefik.

router_instance_pool_id

Description: Instance pool ID of the node pool specifically created for Traefik.

cluster_security_group_id

Description: Security group ID attached to the SKS nodepool instances.

kubernetes_host

Description: Endpoint for your Kubernetes API server.

kubernetes_cluster_ca_certificate

Description: Certificate Authority required to communicate with the cluster.

kubernetes_client_key

Description: Certificate Client Key required to communicate with the cluster.

kubernetes_client_certificate

Description: Certificate Client Certificate required to communicate with the cluster.

raw_kubeconfig

Description: Raw .kube/config file for kubectl access.

Reference in table format

Show tables

= Requirements

Name Version

terraform

>= 1.0

exoscale

>= 0.49

external

>= 2.1

kubernetes

>= 2.21

= Providers

Name Version

exoscale

>= 0.49

local

n/a

= Resources

Name Type

exoscale_anti_affinity_group.this

resource

exoscale_domain_record.wildcard_with_cluster_name

resource

exoscale_nlb.this

resource

exoscale_security_group.this

resource

exoscale_security_group_rule.all

resource

exoscale_security_group_rule.calico_traffic

resource

exoscale_security_group_rule.cilium_health_check

resource

exoscale_security_group_rule.cilium_health_check_icmp

resource

exoscale_security_group_rule.cilium_traffic

resource

exoscale_security_group_rule.http

resource

exoscale_security_group_rule.https

resource

exoscale_security_group_rule.nodeport_tcp_services

resource

exoscale_security_group_rule.nodeport_udp_services

resource

exoscale_security_group_rule.sks_logs

resource

exoscale_sks_cluster.this

resource

exoscale_sks_kubeconfig.this

resource

exoscale_sks_nodepool.this

resource

local_sensitive_file.sks_kubeconfig_file

resource

exoscale_domain.this

data source

= Inputs

Name Description Type Default Required

cluster_name

The name of the Kubernetes cluster to create.

string

n/a

yes

base_domain

The base domain used for Ingresses. If not provided, nip.io will be used taking the NLB IP address.

string

null

no

description

A free-form string description to apply to the SKS cluster.

string

null

no

zone

The name of the zone where to deploy the SKS cluster. Available zones can be consulted here.

string

n/a

yes

kubernetes_version

Kubernetes version to use for the SKS cluster. See exo compute sks versions for reference. May only be set at creation time.

string

n/a

yes

auto_upgrade

Enable automatic upgrade of the SKS cluster control plane.

bool

false

no

service_level

Choose the service level for the SKS cluster. Starter can be used for test and development purposes, Pro is recommended for production workloads. The official documentation is available here.

string

"pro"

no

nodepools

Map containing the SKS node pools to create. Needs to be a map of maps, where the key is the name of the node pool and the value is a map containing at least the keys instance_type and size. The other keys are optional: description, instance_prefix, disk_size, labels, taints and private_network_ids. Check the official documentation here for more information.

 
map(object({
    size                = number
    instance_type       = string
    description         = optional(string)
    instance_prefix     = optional(string, "pool")
    disk_size           = optional(number, 50)
    labels              = optional(map(string), {})
    taints              = optional(map(string), {})
    private_network_ids = optional(list(string), [])
  }))

null

no

router_nodepool

Configuration of the router node pool. The defaults of this variable are sensible and rarely need to be changed. The variable is mainly used to change the size of the node pool when doing cluster upgrades.

 
object({
    size            = number
    instance_type   = string
    instance_prefix = optional(string, "router")
    disk_size       = optional(number, 20)
    labels          = optional(map(string), {})
    taints = optional(map(string), {
      nodepool = "router:NoSchedule"
    })
    private_network_ids = optional(list(string), [])
  })
 
{
  "instance_type": "standard.small",
  "size": 2
}

no

tcp_node_ports_world_accessible

Create a security group rule that allows world access to to NodePort TCP services. Recommended to leave open as per SKS documentation.

bool

true

no

udp_node_ports_world_accessible

Create a security group rule that allows world access to to NodePort UDP services.

bool

false

no

cni

Specify which CNI plugin to use (cannot be changed after the first deployment). Accepted values are calico or cilium. This module creates the required security group rules.

string

"cilium"

no

kubeconfig_ttl

Validity period of the Kubeconfig file in seconds. See official documentation for more information.

number

0

no

kubeconfig_early_renewal

Renew the Kubeconfig file if its age is older than this value in seconds. See official documentation for more information.

number

0

no

create_kubeconfig_file

Create a Kubeconfig file in the directory where terraform apply is run. The file will be named <cluster_name>-config.yaml.

bool

false

no

= Outputs

Name Description

cluster_name

Name of the SKS cluster.

base_domain

The base domain for the SKS cluster.

cluster_id

ID of the SKS cluster.

nlb_ip_address

IP address of the Network Load Balancer.

nlb_id

ID of the Network Load Balancer.

router_nodepool_id

ID of the node pool specifically created for Traefik.

router_instance_pool_id

Instance pool ID of the node pool specifically created for Traefik.

cluster_security_group_id

Security group ID attached to the SKS nodepool instances.

kubernetes_host

Endpoint for your Kubernetes API server.

kubernetes_cluster_ca_certificate

Certificate Authority required to communicate with the cluster.

kubernetes_client_key

Certificate Client Key required to communicate with the cluster.

kubernetes_client_certificate

Certificate Client Certificate required to communicate with the cluster.

raw_kubeconfig

Raw .kube/config file for kubectl access.