Deployment on Amazon EKS
An example of a local deployment on Amazon EKS is provided here. Clone this repository and modify the files at your convenience. In the folder, as in a standard Terraform module, you will find the following files:
-
terraform.tf
- declaration of the Terraform providers used in this project; -
locals.tf
- local variables used by the DevOps Stack modules; -
main.tf
- definition of all the deployed modules; -
s3_loki.tf
- creation of the IAM policy, assumable role and bucket used by Loki; -
s3_thanos.tf
- creation of the IAM policy, assumable role and bucket used by Thanos; -
csi_drivers.tf
- creation of the required resources as well as the DevOps Stack modules needed for the CSI drivers of the cluster; -
outputs.tf
- the output variables of the DevOps Stack;
Requirements
On your local machine, you need to have the following tools installed:
-
Terraform to provision the whole stack;
-
AWS CLI to interact with your AWS account;
-
gopass
andsummon
to easily pass the IAM secrets as environment variables when runningterraform
commands;
Other than that, you will require the following:
-
an AWS account;
-
an AWS IAM key with at least the … … …
-
a Route 53 zone;
Specificities and explanations
secrets.yml
Check this blog post for more information on how to configure gopass and summon to work together.
|
For simplicity and ease of use, as well as security, the example uses gopass
and summon
to pass the IAM credentials to the Terraform commands. The secrets.yml
file contains the path to the the secret values on the gopass
password store. On execution, the summon
command will then read the secrets.yml
file and pass the credentials as environment variables to the Terraform commands.
The commands presented on this tutorial all use the summon
command.
The environment variable AWS_DEFAULT_REGION defines where all the AWS resources created by Terraform will reside, including the EKS cluster.
|
Remote Terraform state
If you do not want to configure the remote Terraform state backend, you can simply remove the backend
block from the terraform.tf
file.
More information about the remote backends is available on the official documentation. |
OIDC authentication
The DevOps Stack modules are developed with OIDC in mind. In production, you should have an identity provider that supports OIDC and use it to authenticate to the DevOps Stack applications. |
In this example, we use the Amazon EKS OIDC provider. We provide a module that takes in a Cognito pool ID and its domain to provide you with the required configuration to deploy the DevOps Stack applications.
This assumes that you have created a Cognito pool yourself, however you can use our module to also create the pool and populate it with users, as shown in the example.
Check the AWS Cognito OIDC usage documentation for more information on how to use it. |
The user_map
variable of that module allows you to create OIDC users used to authenticate to the DevOps Stack applications. You should receive an e-mail from AWS with a temporary password to login for the first time.
Let’s Encrypt SSL certificates
By default, to avoid rate-limiting your domain by Let’s Encrypt, the example uses the letsencrypt-staging
configuration of the cert-manager module to generate certificates using the Let’s Encrypt staging environment which has an invalid CA certificate.
If you feel ready to test with production certificates, you can simply edit the locals.tf
file and change the cluster_issuer
variable to letsencrypt-prod
.
Deployment
-
Clone the repository and
cd
into theexamples/eks
folder; -
Adapt the
secrets.yml
file to point to the correct path on yourgopass
password store; -
Check out the modules you want to deploy in the
main.tf
file, and comment out the others;You can also add your own Terraform modules in this file or any other file on the root folder. A good place to start to write your own module is to clone the devops-stack-module-template repository and adapt it to your needs. -
On the
oidc
module, adapt theuser_map
variable as you wish (please check the OIDC section for more information). -
From the source of the example deployment, initialize the Terraform modules and providers:
summon terraform init
-
Configure the variables in
locals.tf
to your preference:The cluster_name
andvpc_cidr
must be unique for each DevOps Stack deployment in a single AWS account and thebase_domain
must match a Route 53 zone in that same account.The cluster module documentation can help you know what to put in the kubernetes_version
, for example.locals { kubernetes_version = "1.28.1" cluster_name = "YOUR_CLUSTER_NAME" zone = "YOUR_CLUSTER_ZONE" service_level = "starter" base_domain = "your.domain.here" activate_wildcard_record = true cluster_issuer = "letsencrypt-staging" enable_service_monitor = false # Can be enabled after the first bootstrap. app_autosync = true ? { allow_empty = false, prune = true, self_heal = true } : {} }
-
Finally, run
terraform apply
and accept the proposed changes to create the Kubernetes nodes on Amazon EKS and populate them with our services;summon terraform apply
-
After the first deployment (please note the troubleshooting step related with Argo CD), you can go to the
locals.tf
and enable the ServiceMonitor boolean to activate the Prometheus exporters that will send metrics to Prometheus;This flag needs to be set as false
for the first bootstrap of the cluster, otherwise the applications will fail to deploy while the Custom Resource Definitions of the kube-prometheus-stack are not yet created.You can either set the flag as true
in thelocals.tf
file or you can simply delete the line on the modules' declarations, since this variable is set astrue
by default on each module.Take note of the local called app_autosync
. If you set the condition of the ternary operator tofalse
you will disable the auto-sync for all the DevOps Stack modules. This allows you to choose when to manually sync the module on the Argo CD interface and is useful for troubleshooting purposes.
Access the cluster and the DevOps Stack applications
To access your cluster you need to use the AWS CLI to recover a Kubeconfig you can use:
summon aws eks update-kubeconfig --name YOUR_CLUSTER_NAME --region YOUR_CLUSTER_ZONE --kubeconfig ~/.kube/NAME_TO_GIVE_YOUR_CONFIG.config
Then you can use the kubectl
or k9s
command to interact with the cluster:
k9s --kubeconfig ~/.kube/NAME_TO_GIVE_YOUR_CONFIG.config
As for the DevOps Stack applications, you can access them through the ingress domain that you can find in the ingress_domain
output. If you used the code from the example without modifying the outputs, you will see something like this on your terminal after the terraform apply
has done its job:
Outputs:
devops_admins = <sensitive>
ingress_domain = "your.domain.here"
Or you can use kubectl
to get all the ingresses and their respective URLs:
kubectl get ingress --all-namespaces --kubeconfig ~/.kube/NAME_TO_GIVE_YOUR_CONFIG.config
The output devops_admins
list all users and respective e-mails that were configured using the OIDC module:
summon terraform output devops_admins
Those users should have received an e-mail with a temporary password in order to login to the DevOps Stack applications for the first time.
Stop the cluster
To definitively stop the cluster on a single command (that is the reason we delete some resources from the state file), you can use the following command:
summon terraform state rm $(summon terraform state list | grep "argocd_application\|argocd_project\|kubernetes_\|helm_") && summon terraform destroy
Conclusion
That’s it, you now have a fully functional Kubernetes cluster in Amazon EKS with the DevOps Stack applications deployed on it. For more information, keep on reading the documentation. You can explore the possibilities of each module and get the link to the source code on their respective documentation pages.
Troubleshooting
connection_error
during the first deployment
In some cases, you could encounter an error like these the first deployment:
╷
│ Error: error while waiting for application argocd to be created
│
│ with module.argocd.argocd_application.this,
│ on .terraform/modules/argocd/main.tf line 55, in resource "argocd_application" "this":
│ 55: resource "argocd_application" "this" {
│
│ error while waiting for application argocd to be synced and healthy: rpc error: code = Unavailable desc = error reading from server: EOF
╵
The error is due to the way we provision Argo CD on the final steps of the deployment. We use the bootstrap Argo CD to deploy the final Argo CD module, which causes a redeployment of Argo CD and consequently a momentary loss of connection between the Argo CD Terraform provider and the Argo CD server.
You can simply re-run the command summon terraform apply
to finalize the bootstrap of the cluster every time you encounter this error.
Argo CD interface reload loop when clicking on login
If you encounter a loop when clicking on the login button on the Argo CD interface, you can try to delete the Argo CD server pod and let it be recreated.
For more informations about the Argo CD module, please refer to the respective documentation page. |