This repo contains a Terraform modules for creating Kubernetes clusters on Google Cloud Platform (GCP), Amazon Web Services(AWS), Microsoft Azure, Tanzu.
- gcp: Google cloud terraform module to create kubernetes cluster using Google Kubernetes Engine(GKE).
- aws: Amazon Web Services terraform module to create kubernetes cluster using Elastic Kubernetes Service(EKS).
- azure: Azure terraform module to create kubernetes cluster using Azure Kubernetes Service(AKS).
- tanzu: Tanzu terraform module to create kubernetes cluster using Elastic Kubernetes Service.
- test: Automated tests for the files in this project repository.
Kubernetes is an open source container management system for deploying, scaling, and managing containerized applications. Kubernetes is built by Google based on their internal proprietary container management systems (Borg and Omega). Kubernetes provides a cloud agnostic platform to deploy your containerized applications with built in support for common operational tasks such as replication, autoscaling, self-healing, and rolling deployments.
Managed Kubernetes is when third-party providers take over responsibility for some or all of the work necessary for the successful set-up and operation of K8s. Depending on the vendor, “managed” can refer to anything from dedicated support, to hosting with pre-configured environments, to full hosting and operation. We will be using GKE, AKS, EKS, Tanzu.
Terraform is a tool for building, changing, and versioning infrastructure safely and efficiently. Terraform can manage existing and popular service providers as well as custom in-house solutions. We will be using terraform version 0.14 to provision out infrastructure.
A Terraform Module is a canonical, reusable, best-practices definition for how to run a single piece of infrastructure, such as a database or server cluster. Each Module is written using a combination of Terraform and scripts (mostly bash) and include automated tests, documentation, and examples. It is maintained both by the open source community and companies that provide commercial support.
Instead of figuring out the details of how to run a piece of infrastructure from scratch, you can reuse existing code that has been proven in production. And instead of maintaining all that infrastructure code yourself, you can leverage the work of the Module to pick up infrastructure improvements through a version number bump.
At a minimum 61 GiB of memory, 8 vCPUs virtual machines are needed to run k8ssandra. Minimum recommendation for volumes is 1.5 - 2 TB, but that's all set up through the persistent volume requests.
-
Naming Conventions: All the resources will be created with the prefix of
environment-project_name.- eg: environment="development" and Project_name="k8ssandra" resource_name "development-k8ssandra-gke-cluster"
-
Naming Limitation: Every cloud provider have limitations on the resource names, they will only allow resource names up to some characters long.
- eg: If we pass
environment=production theproject_name=K8ssandra-terraform-project-resources-for-multiple-cloud-providers your resource will create as resource_name =production-K8ssandra-terraform-project-resources-for-multiple-cloud-providers-gke-cluster
-
In the above example the resource name exceeds more than 63 characters long. It is an invalid resource name, these will error out when you run
Terraform planorTerraform validatecommands. These limitations are hard limitations which can not be changed by your cloud provider. make sure you followed naming standards while creating your resources. It is a good practice maintain limits on length of resource names. -
refer the following documentation
- eg: If we pass
k8ssandra-terraform/ | aws/ | gcp/ | ├──modules/ | | ├──gcs | | ├── main.tf | | └── variables.tf | | └── outputs.tf | | └── README.md | | ├──vpc | | ├── main.tf | | └── variables.tf | | └── outputs.tf | | └── README.md | | ├──iam | | ├── main.tf | | └── variables.tf | | └── outputs.tf | | └── README.md | | ├──gke | | ├── main.tf | | └── variables.tf | | └── outputs.tf | | └── README.md | └──README.md | └──gitignore | ├──env | ├── dev.tf | ../modules/vpc | ../modules/iam | ../modules/gke_cluster | ├── version.tf | └── backend.tf | └── variables.tf | └── outputs.tf | └── README.md | ├──scripts | azure/ | tanzu/ | test/ | LICENSE | Makefile | README.md
- Terraform scripts are in different folders choose your cloud provider to create resources in.
list of available providers "aws, gcp, azure "
make helpTo list out the available options to use.make init "provider=<REPLACEME>"Initialize and configure Backend.make plan "provider=<REPLACEME>"Plan all Terraform resources.make apply "provider=<REPLACEME>"Create or update Terraform resources.make destroy "provider=<REPLACEME>"Destroy all Terraform resources.make lintCheck syntax of all scripts.make getpodsGet running pods IPs and Namespaces run this command after apply
-
Testing this project Locally gcp
-
Set up environment on your machine before running the make commands. use the following links to setup your machine.
-
How to create GKE cluster resources by using the make command Before using the make commands export the following terraform environment variables(TFVARS) for terraform to create the resources.
export TF_VAR_environment=<ENVIRONMENT_REPLACEME>
ex:- export TF_VAR_environment=dev
export TF_VAR_name=<CLUSTERNAME_REPLACEME>
ex:- export TF_VAR_name=k8ssandra
export TF_VAR_project_id=<PROJECTID_REPLACEME>
ex:- export TF_VAR_project_id=k8ssandra-testing
export TF_VAR_region=<REGION_REPLACEME>
ex:- export TF_VAR_region=us-central-1
#To list out the available options to use.
make helpimportant: Before running the following command, we need to Export the environment variables as show above.
# Initialize and configure Backend.
make init "provider=gcp"# Plan all GCP resources.
make plan "provider=gcp"# Create or update GCP resources
# This command takes some time to execute.
make apply "provider=gcp"# Destroy all GCP resources
make destroy "provider=gcp"-
Testing this project Locally AWS EKS
-
Set up environment on your machine before running the make commands. use the following links to setup your machine.
-
How to create AWS EKS cluster resources by using the make command Before using the make commands export the following terraform environment variables(TFVARS) for terraform to create the resources.
# Environment name, eg. "dev"
# bash, zsh
export TF_VAR_environment=dev
#fish
set -x TF_VAR_environment dev
# Kubernetes cluster name, eg. "k8ssandra"
# bash, zsh
export TF_VAR_name=k8ssandra
# Resource Owner name, eg. "k8ssandra"
# bash, zsh
export TF_VAR_resource_owner=k8ssandra
#fish
set -x TF_VAR_resource_owner k8ssandra
#fish
set -x TF_VAR_name k8ssandra
# AWS region name, eg. "us-east-1"
# bash, zsh
export TF_VAR_region=us-east-1
#fish
set -x TF_VAR_region us-east-1
#To list out the available options to use.
make helpimportant: Before running the following command, we need to Export the environment variables as show above.
# Initialize and configure Backend.
make init "provider=aws"# Plan all GCP resources.
make plan "provider=aws"# Create or update AWS resources
# This command takes some time to execute.
make apply "provider=aws"# Destroy all AWS resources created
make destroy "provider=aws"-
Testing this project Locally azure
-
Set up environment on your machine before running the make commands. use the following links to setup your machine.
-
How to create AKS cluster resources by using make command Before using the make commands export the following terraform environment variables(TFVARS) for terraform to create the resources.
export TF_VAR_environment=<ENVIRONMENT_REPLACEME>
ex:- export TF_VAR_environment=dev
export TF_VAR_name=<CLUSTERNAME_REPLACEME>
ex:- export TF_VAR_name=k8ssandra
export TF_VAR_region=<REGION_REPLACEME>
ex:- export TF_VAR_region=eastus
#To list out the available options to use.
make helpimportant: Before running the following command, we need to Export the environment variables as show above.
# Initialize and configure Backend.
make init "provider=azure"# Plan all Azure resources.
make plan "provider=azure"# Create or update Azure resources
# This command will take some time to execute.
make apply "provider=azure"# Destroy all Azure resources created with terraform.
make destroy "provider=azure"-
The create script fails with a
Permission deniedwhen running Terraform - The credentials that Terraform is using do not provide the necessary permissions to create resources in the selected projects. Ensure that the account listed ingcloud config listhas necessary permissions to create resources. If it does, regenerate the application default credentials usinggcloud auth application-default login. -
Terraform timeouts - Sometimes resources may take longer than usual to create and Terraform will timeout. The solution is to just run
make createagain. Terraform should pick up where it left off. -
Terraform state lock - Sometime if two are more people working on the same Terraform state file a lock will be placed on your remote Terraform state file, to unlock the state run the following command
terraform force-unlock <LOCK_ID>. -
Terraform Incomplete resource deletion - If you created some resources manually on the cloud console and attach those resources to the resources created by the Terraform,
terraform destroyormake destroycommands will fail. To resolve those errors you will have to login into the cloud console, delete those resource manually and runmake destroyorterraform destory.