🌟HashiCorp Vault cluster on AWS with Auto Unseal and TLS🌟

This module uses two powerful IaC tools - Terraform and Ansible, to deploy and manage a robust HashiCorp Vault environment on AWS, focusing on simplicity, security, and high availability.

📋 Overview

• 🚀 Easy Deployment: Automatically sets up your HashiCorp Vault cluster in AWS, ensuring the process is quick and straightforward.
• 🛡️ High Availability: The cluster operates across two Availability Zones (AZs) within AWS, featuring a parameterized number of bastion hosts in a single public subnet and Vault instances distributed across two private subnets for fault tolerance and resilience.
• ⚙️ Dynamic Configuration⚙: Using subnet_id = element(var.private_subnet_ids, count.index % length(var.private_subnet_ids)), Vault instances are intelligently distributed across available AZs.

🌐 Terraform's Role

Terraform is responsible for setting up all the required infrastructure on AWS and preparing the environment for Vault:

1. Infrastructure Creation: Constructs all required AWS infrastructure components, including EC2 instances, networking setups, and security groups.
2. TLS Management: Generates self-signed TLS certificates — a single root CA (ca.crt) and unique tls-cert and tls-key for each Vault node to secure communications.
3. Configuration Files: Produces unique vault.hcl configuration files for each node, including settings for AWS KMS auto unseal.
4. Static Files: Creates essential static files like vault.service for systemd management and ansible.cfg to standardize Ansible's execution.
5. Dynamic Inventory: Dynamically generates an Ansible inventory, adding essential parameters for Ansible roles to enable precise configuration management.

🌐 Ansible's Role

Ansible takes over the configuration management to ensure that the Vault operates correctly and securely:

1. Vault Setup: Installs and configures Vault using the binary distribution on each node.
2. Configuration Distribution: Distributes the correct vault.hcl files to each node using the dynamic inventory, ensuring each node is configured uniquely and correctly.
3. TLS Configuration: Manages and distributes TLS certificates to each node, aligning with the certificates generated by Terraform.

Dynamic Inventory Example

resource "local_file" "ansible_inventory_yaml" {      # Simple Dynamic Inventory for Ansible
  content = yamlencode({
    all = {
      children = {
        vault = {
          vars = {
            ansible_ssh_common_args = "-o ProxyCommand='ssh -o StrictHostKeyChecking=no -W %h:%p -q -i ~/.ssh/${var.ssh_key_name} ubuntu@${var.bastion_public_ip[0]}'"
          }
          hosts = {
            for index, ip in var.vault_instance_private_ip : "vault${index + 1}" => { 
              ansible_host                 = ip
              ansible_user                 = "ubuntu"
              ansible_ssh_private_key_file = pathexpand("~/.ssh/${var.ssh_key_name}")            
              vault_config_file            = "vault-config-${index + 1}.hcl"        # crucial for Ansible
              tls_cert_file                = "vault-tls-cert-${index + 1}.pem"      # crucial for Ansible
              tls_key_file                 = "vault-tls-key-${index + 1}.pem"       # crucial for Ansible
            }
          }
        }
      }
    }
  })
  filename = "./AnsibleCode/hosts.yml"
}

Reflection on Idempotency

Yes, the necessity for Ansible role to have specific parameters (vault_config_file, tls_cert_file, tls_key_file) in the inventory file violates the principle of idempotency. However, despite this negative aspect, this approach proved to be the best solution. After all — are these tools created for us, or is it us — for the tools?

🔥 Required Variables and Configuration 🔥

Before deploying the HashiCorp Vault, ensure you configure the following settings to align with your AWS environment and security requirements:

Configuration Details

Type	Setting	Description	Action
Must-Have Configurations
🎯 Dynamic (var)	ssh_key_name	Specify the SSH key name that is region-specific and configured for your AWS account. The region is dynamically detected in the main.tf using data.aws_region.	Specify at runtime: terraform apply -var "ssh_key_name=your-ssh-keyname"
🎯 Dynamic (var)	certificate_arn_lb	TLS certificate ARN for the Application Load Balancer (ALB). This is critical for HTTPS traffic management to your services.	Specify at runtime: terraform apply -var "certificate_arn_lb=your-cert-arn"
🔧 Manual (template)	vault.hcl.tpl	Enter the ARN of your AWS KMS key in the Vault configuration template. This key is used for the auto unseal feature.	Add ARN of your AWS KMS key directly in template: /modules/ansible_templates_generator/templates/vault.hcl.tpl
🔧 Manual (root level)	backend.tf	Specify your own S3 bucket where the vault-nice-cluster.tfstate will be stored. This setup is critical for managing the state of your Terraform deployments and ensuring data persistence across sessions.	Update the file: backend.tf
Optional Configurations
🔄 Default (var)	vpc_cidr	VPC module. This is the base CIDR from which subnet CIDRs are derived using formulas. Defaults are 10.100.0.0/16, Public Subnets: Formula: cidr_block = cidrsubnet(var.vpc_cidr, 8, 10 + count.index) Private Subnets: Formula cidr_block = cidrsubnet(var.vpc_cidr, 8, count.index + 100)	Defaults are set, but can be overridden at runtime: terraform apply -var "vpc_cidr=your-vpc-cidr"
🔄 Default (var)	bastion_ec2_count	Number of EC2 instances for the bastion host. Bastion hosts provide secure access to your private network from the internet.	Default value is 1, but can be overridden at runtime.
🔄 Default (var)	vault_ec2_count	Number of EC2 instances for running Vault servers. This setup is used to manage the high availability and scalability of Vault within AWS.	Default value is 3, but can be overridden at runtime.

Please ensure must-have settings are properly configured in your Terraform and Ansible files to facilitate a smooth and secure deployment.

🔥 Good to Know 🔥

Once you have configured your backend.tf and specified your AWS KMS key in vault.hcl.tpl, you are almost ready to go. Follow these steps to complete the setup:

1. Run Terraform:

   terraform apply -var "ssh_key_name=ssh-keyname"
   

2. Run Ansible: chdir AnsibleCode, then:

   ansible-playbook playbook
   

After these steps, you will need to manually initialize the Vault on one of the nodes (3 by default). To do this, Terraform will provide a command similar to the following for each Vault instance:

3. vault operator init:

ssh -i ~/.ssh/ssh -o ProxyCommand='ssh -W %h:%p -i ~/.ssh/ssh [email protected]' [email protected]`

Once connected - initialize the Vault vault operator init Finally, restart the Vault on each node to form a cluster:

4. Form the cluster:

ansible vault -m shell -a 'sudo systemctl restart vault'

(which is also printed by terraform)

🌍 Planned Enhancements 🌍

🤖 Here are some potential improvements:

• User Flexibility: Enable configuration of user names for Vault and bastion instances beyond the default ubuntu.

• Subnet Configuration: Add opportunity to dynamically specify through variable the number of private and public subnets.

• VPC Endpoint: Add VPC endpoints to provide an extra layer of security within AWS.

• Load Balancer for Health Checks: Plan to implement a load balancer primarily to make monitoring of health checks easier.

• Consul: Implementate Consul

In few words?

• DNS: Set up a CNAME record to direct traffic to the load balancer (lb dns name will be printed by Terraform)
• Application Load Balancer: Manages traffic, health checks and hides the Vault cluster behind a «DNS shield»
• Bastion host/s: The only access point for SSH connections to the Vault nodes, configured via Terraform. (those ssh commands also will be printed by Terraform)
• Vault Cluster: Distributed across private subnets, equipped with auto-unseal and TLS for secure operations.