This project deploys Kafka and part of it's ecosystem to AWS through terraform.
It is mainly developed to help people that want to learn/explore the technology, focused - but not limited - to stream database events.
What is included:
- Kafka Core - AWS MSK
- Kafka Components - AWS ECS
- Kafka Connect (debezium, by
https://debezium.io/
through https://hub.docker.com/r/debezium/connect) - Kafka Connect UI (by
Landoop
through https://hub.docker.com/r/landoop/kafka-connect-ui) - Kafka Schema Registry (by
Confluent
through https://hub.docker.com/r/confluentinc/cp-schema-registry/tags/) - Kafka Schema Registry UI (by
Landoop
through https://hub.docker.com/r/landoop/schema-registry-ui/) - Kafka Rest Proxy API (by
Confluent
through https://hub.docker.com/r/confluentinc/cp-kafka-rest) - Kafka KSQL Server (by
Confluent
through https://hub.docker.com/r/confluentinc/cp-ksql-server)
- Kafka Connect (debezium, by
- VPC
- RDS Database
- Load Balancer
You need the following to deploy the project:
-
AWS account https://aws.amazon.com/premiumsupport/knowledge-center/create-and-activate-aws-account/
- Create an S3 bucket to store terraform state files https://docs.aws.amazon.com/AmazonS3/latest/gsg/CreatingABucket.html
- Create EC2 key pair https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html -
Install Make (optional but recommended). Choose one of the following:
- Using chocolatey (package manager): https://chocolatey.org/packages/make
- Manual installation: http://gnuwin32.sourceforge.net/packages/make.htm -
Install Terraform:
- Using chocolatey (package manager): https://chocolatey.org/packages/terraform
- Manual installation: https://www.terraform.io/downloads.html -
AWS CLI installed with a profile set on your machine:
- Using chocolatey (package manager): https://chocolatey.org/packages/awscli
- Manual installation: https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html
IMPORTANT: After installing it please configure a profile: https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html#cli-quick-configuration
Before you deploy anything you need to make sure variables are set as expected on aws-infrastructure\configuration
.
There are 5 configuration files where most variables are set with default values. However, there is a section on top of each file called NEED CONFIGURATION that needs your manual configuration.
Position on the root level of this project
Deploy:
Using Make
-
To deploy all of terraform
make deploykad
-
To deploy VPC
make deployvpc
-
To deploy MSK
make deploymsk
-
To deploy ECS
make deployecs
-
To deploy RDS
make deployrds
Manual Terraform
-
To deploy VPC
cd aws-infrastructure/deployment/kad-vpc terraform init -backend-config="../../configuration/terraform-backend.tfvars" terraform apply -auto-approve -var-file="../../configuration/global-config.tfvars"
-
To deploy MSK
cd aws-infrastructure/deployment/kad-msk terraform init -backend-config="../../configuration/terraform-backend.tfvars" terraform apply -auto-approve -var-file="../../configuration/global-config.tfvars" -var-file="../../configuration/msk-config.tfvars"
-
To deploy ECS
cd aws-infrastructure/deployment/kad-ecs terraform init -backend-config="../../configuration/terraform-backend.tfvars" terraform apply -auto-approve -var-file="../../configuration/global-config.tfvars" -var-file="../../configuration/ecs-config.tfvars"
-
To deploy RDS
cd aws-infrastructure/deployment/kad-rds terraform init -backend-config="../../configuration/terraform-backend.tfvars" terraform apply -auto-approve -var-file="../../configuration/global-config.tfvars" -var-file="../../configuration/rds-config.tfvars"
Destroy:
Using Make
-
To destroy all of terraform
make destroykad
kad = kafka aws deployment
-
To destroy RDS
make destroyrds
-
To destroy ECS
make destroyecs
-
To destroy MSK
make destroymsk
-
To destroy VPC
make destroyvpc
Manual Terraform
-
To destroy RDS
cd aws-infrastructure/deployment/kad-rds terraform init -backend-config="../../configuration/terraform-backend.tfvars" terraform destroy -auto-approve -var-file="../../configuration/global-config.tfvars" -var-file="../../configuration/rds-config.tfvars"
-
To destroy ECS
cd aws-infrastructure/deployment/kad-ecs terraform init -backend-config="../../configuration/terraform-backend.tfvars" terraform destroy -auto-approve -var-file="../../configuration/global-config.tfvars" -var-file="../../configuration/ecs-config.tfvars"
-
To destroy MSK
cd aws-infrastructure/deployment/kad-msk terraform init -backend-config="../../configuration/terraform-backend.tfvars" terraform destroy -auto-approve -var-file="../../configuration/global-config.tfvars" -var-file="../../configuration/msk-config.tfvars"
-
To destroy VPC
cd aws-infrastructure/deployment/kad-vpc terraform init -backend-config="../../configuration/terraform-backend.tfvars" terraform destroy -auto-approve -var-file="../../configuration/global-config.tfvars"
Important Notes:
The architecture is designed for Kafka exploration and not a proper production deployment.
- The database is placed on a public subnet to allow interaction with RDS through external database clients such as DBeaver, pgAdmin.
- An autoscaling group is in place to facilitate the addition of new containers without worrying about the underlying infrastructure. You can change min, max and desired number of instances on this file
aws-infrastructure\configuration\ecs-config.tfvars