How should I create the backend (e.g., S3 bucket) for storing Terraform state?

[brikis98]

We get the following question from customers fairly often, so I'll post the question here, and then add an answer below:

How should I create the backend (e.g., S3 bucket) for storing Terraform state?

---

Tracked in ticket #110450

[brikis98]

There are several different ways to handle this, each with various pros and cons:

Option 1: create the backend manually (ClickOps)

Overview

One option is to create your backend manually. For example, if you're using S3 as a backend, you'd login to the AWS Console, and click around for a while to create an S3 bucket and DynamoDB table.

Pros

• It's an easy way to get started. If you only have a small number of backends to manage (e.g., your whole company just needs 1-3 S3 buckets), this approach works just fine.
• You avoid the "chicken & egg" problem described below.
• You can configure your backend however you want.

Cons

• It's a manual process, so if you need more backends in the future, and/or you have a lot of backends to manage (e.g., dozens or hundreds of S3 buckets), this approach does not scale well.
• You're not managing all your infrastructure as code.
• How you set up the backend is not documented, so if you need to do it again the future, it won't be obvious to your team members what to do.
• It's easy to make mistakes while doing ClickOps: e.g., you might forget to turn on an important security setting for your S3 bucket.

Option 2: create the backend using Terraform

Overview

Another option is to write a Terraform module to set up your backend. For example, if you're a Gruntwork customer, you can use the s3-bucket module to create a secure, best-practices S3 bucket. As for the DynamoDB table, it's a single resource (example).

The chicken & egg problem

One catch here is that you have a bit of a chicken & egg problem: you want to store all your Terraform state in a backend, but here, the Terraform code you're deploying is what creates the backend, which doesn't exist initially, so you can't store state in it. Working around this is pretty simple, but does involve some manual steps:

1. Run init and apply on your backend module without configuring any sort of backend in your code.
2. When apply completes, your backend will be set up (e.g., the S3 bucket and DynamoDB table will now exist), and the state file for your backend module will be sitting on your local disk drive (terraform.tfstate).
3. At this point, configure a backend block for your backend module to store its state in the backend you just created.
4. Run init again. Terraform will prompt you to migrate the local state file (terraform.tfstate) into your backend. Say, "yes."

Note that you have to do these steps before you can run apply on any other modules in this environment, as those modules need this backend to be set up first. So this approach involves some ordering dependencies when setting up new environments.

Pros

• If you can automate this process (see "Cons" below), the solution scales reasonably well.
• Everything is managed as code.
• The code acts as documentation for how the backend is configured, so future team members will be able to figure out how to set up backends for new environments.
• You are using code to manage your backend configuration, so you avoid manual errors.
• You can configure your backend however you want.

Cons

• It's harder to get started, as you have to write code and set up processes.
• You have the chicken and egg problem: a series of manual steps to create backend and then have the backend module use the backend it just created.
• You have to enforce ordering, ensuring that the backend module is the first thing deployed in each new environment.

Option 3: let Terragrunt create your backend

Overview

For a few popular backend types (S3 and GCS), if you're using the remote_state block, Terragrunt can automatically create the backend for you. That is, when you run terragrunt init or terragrunt apply, Terragrunt will check if the backend exists, and if it doesn't, it'll make API calls to set up the backend automatically / transparently, following best practices.

Pros

• It's an easy way to get started, as Terragrunt handles the backend set up for you.
• It's a fully automated process and it scales well.
• Everything is managed as code (the remote_state block configures your backend).
• The code acts as documentation for how the backend is configured (along with Terragrunt's docs), so future team members will be able to figure out how to set up backends for new environments.
• You are using code to manage your backend configuration, so you avoid manual errors.
• You avoid the chicken & egg problem described above. Even ordering isn't a problem, as Terragrunt handles locking to deal with concurrency (e.g., if two modules try to use the same backend that doesn't exist at the same time, one will create it, and the other will wait).

Cons

• Not all backend types are supported.
• You have some ability to customize the backend configuration, and Terragrunt does try to follow best practices by default, but you don't have as much control as when you set up the backend yourself using ClickOps or a Terraform module.
• Terragrunt currently doesn't have a built in way to update existing backends. So creation is handled automatically, but if later on, you need to evolve your backend approach, that's more of a manual process.

Option 4: use a SaaS managed backend

Overview

Some SaaS tools for Terraform, such as Terraform Cloud and Terraform Enterprise, manage backend for you "magically." That is, you don't have to think about the backend at all: the SaaS tool just does it automatically, transparently, behind the scenes.

Pros

• It's an easy way to get started, as the SaaS tool handles the backend set up for you.
• It's a fully automated process and it scales well.
• You don't really have documentation for how to set up new backends, but you don't entirely need it, as the SaaS does it magically.
• The SaaS tool does all the work for you automatically, so you avoid manual errors.
• You avoid the chicken & egg problem described above.

Cons

• You aren't really managing things as code. They are just managed magically by the SaaS.
• For the most part, you can't customize how your state is stored. You just have to trust the SaaS tool is doing the right thing. Note that state files contain secrets, so you are putting a lot of trust in this SaaS tool.
• Lock in. Migrating away from the SaaS can be tricky.
• Migrating state is a bit tricky.

[giner]

There is a simpler module for option 2 which handles both s3 bucket and dynamo db, supports multiple states and creates backend configs automatically https://registry.terraform.io/modules/ansraliant/s3-state/aws/latest

[thiagolsfortunato]

I've always used Option 2, I like to control the bucket creation, for it I already have a module that setup everything that I need. For me, the states are the most important part of the Terraform.

[RafaelHGBotelho]

I use Option 1 in simple cases, and Option 2 in all other cases, BUT I understand that Option 2 is the most correct in all cases.

[lorengordon]

For options 1 & 3, I'd enhance them by adding a terraform config that matches the backend, with import blocks to bring the pre-created resources into tfstate. I think it's particularly powerful in combination with option 3... no more chicken-egg problem!

[dmfigol]

Another option, which would be an extension of the option 1 is to create the bucket via cloud native tool, e.g. AWS CloudFormation/CDK

[NIXKnight]

I detailed my way of setting up the backend through Terragrunt and Terraform using option 2 in this blog post while learning how to use Terragrunt. This contains the initial set of steps. This way you get to set the state bucket parameters with Terraform.

[ansromanov]

For some past projects, mostly done with the bare Terraform, I've preferred to use AWS Cloudformation for the initial S3 bucket and DynamoDB lock table creation. It allowed me to keep the IaC approach without having headaches about local state management.

For newer AWS projects where I have to use Terragrunt wherever possible, I prefer to let Terragrunt manage S3 state bucket creation. But I still have a dedicated Terraform module that sets the correct access permissions to the state bucket as a part of my general Terragrunt live infrastructure.

So, I vote for Option 3.

[sowinski]

Can you show us an example how you do option 3 for AWS?

Unfortunately I get an error:

terragrunt init
Remote state S3 bucket terraform-state-rfer2345 does not exist or you don't have permissions to access it. Would you like Terragrunt to create it? (y/n) y

Initializing the backend...
╷
│ Error: Invalid backend configuration argument
│ 
│ The backend configuration argument "create_bucket" given on the command
│ line is not expected for the selected backend type.
╵

ERRO[0019] terraform invocation failed in /home/ps/Dokumente/git/monteurzimmer.guru/terraform/dev 
ERRO[0019] 1 error occurred:

This is my terragrunt.hcl file

remote_state {
  backend = "s3"

  config = {
    bucket         = "terraform-state-rfer2345"
    key            = "${path_relative_to_include()}/terraform.tfstate"
    region         = "eu-central-1"
    encrypt        = true
    dynamodb_table = "terraform-locks-rfer2345"
    create_bucket  = true
  }
}

terragrunt version v0.52.7
Terraform v1.6.2
on linux_amd64

[arukaen]

Interestingly I'm getting the same error on a bucket that is/was in use for staging and production envs with no problems last week.

[MozeBaltyk]

What about if I want to deploy with github workflow and terraform with backend in S3 ? then once all the pipeline is finish delete the bucket ?

[celloist]

I'm struggling with this too. Coming from Azure you would specify it in your Terraform init and it automaticaly creates the resource group with a storage for the tf state. Found it weird that its not the same for AWS

[MozeBaltyk]

I manage by creating a bucket thanks to s3cmd github actions:

jobs:
  backend:
    name: Backend
    runs-on: ubuntu-latest

    steps:
      - name: Set up S3cmd cli tool
        uses: s3-actions/s3cmd@main
        with:
          provider: digitalocean
          region: FRA1
          access_key: ${{secrets.DIGITALOCEAN_SPACES_ACCESS_TOKEN}}
          secret_key: ${{secrets.DIGITALOCEAN_SPACES_SECRET_KEY}}

      - name: Create Space Bucket
        run: |
          sed -i -e 's/signature_v2.*$/signature_v2 = True/' ~/.s3cfg
          buck="github-action-${{ github.run_id }}"
          s3cmd mb s3://$buck
          sleep 10
        continue-on-error: true

You can also do it manually:

# install 
sudo pip install s3cmd

# config 
s3cmd --configure 

# Create if does not already exist
s3cmd info s3://terraform-backend-test || s3cmd mb s3://terraform-backend-test -v -d

then use this bucket in terraform:

terraform {
  required_providers {
    digitalocean = {
      source = "digitalocean/digitalocean"
      version = "~> 2.0"
    }
  }
  backend "s3" {
    skip_region_validation      = true
    skip_credentials_validation = true
    skip_metadata_api_check     = true
    skip_requesting_account_id  = true
    use_path_style              = true
    skip_s3_checksum            = true
    endpoints = {
      s3 = "https://fra1.digitaloceanspaces.com"
    }
    region                      = "fra1"
    // bucket                   = "terraform-backend-github"
    key                         = "terraform.tfstate"
  }
}