How to configure Gruntwork Pipelines for a setup based on Terragrunt live examples

[zackproser]

A customer asked:

We've based our current setup on https://github.com/gruntwork-io/terragrunt-infrastructure-live-example - is this close enough to get Gruntwork Pipelines working without having a Reference Architecture deployed for us?

[zackproser]

Given your setup and your prior experience with Terragrunt, we recommend that you start by adopting service catalog modules into your current setup. This will likely end up looking like an incremental migration moving component by component. Here's how we'd recommend approaching this:

1. Start by looking at the for-production examples in the service catalog
2. Pick out a service that will be easy to adopt given your current architecture. For example, you could run terragrunt graph-dependencies and pick a component from a leaf-node level to begin with. That is, start with a component that has no downstream dependencies that have not already been migrated.
3. Back up your current terraform state for that module using terragrunt state push backup.tfstate. This is important! If for any reason the migration of state objects with terragrunt state mv commands fails, you can then use terragrunt state pull > backup.tfstate to restore your original state and try again.
4. Replace the terraform source to point at our service catalog module instead of whatever you're using now.
5. Run terragrunt plan. Look at the plan output to see what exactly would change if you were to switch over to using Gruntwork's service catalog modules. Before making any state migrations, try to map destroyed resources to newly created resources. You may be able to match a destroyed resource to a newly created one, and avoid the destroy/create process by using terragrunt state mv.
6. Your goal is to get to a point where the terragrunt plan output is clean with no changes to be applied. But in your case, you will very likely end up with changes that must be made, since we have no idea what modules and module versions you're starting from. The migration of resources can also end up being difficult if you have resources stored in different object types than what our service catalog modules expect. This may require you to write some logic that will migrate objects into different types.
   • Take a look at the _scripts/migration_helpers.sh file in this release: (https://github.com/gruntwork-io/infrastructure-live-multi-account-acme/releases/download/v0.0.1-20210527/refarch-folder-structure.zip). This script contains utilities that perform more complex migrations. Please be aware that we will not support fixes or patches to this frozen artifact. It's offered in this answer only as an example.

Following this pattern, you'd ideally be switching out your current sources in your terragrunt configs for our service catalog source URLs, and continue running plan and doing state migrations until the plan comes out clean or almost clean.

Specifically for the Gruntwork Pipelines piece, if you get the Reference Architecture deployed, then everything "just works". While we don't believe the exact folder structure should matter much, you may need to tweak some of the configuration once you have pipelines set up in order to accommodate differences in your layout.

The good news in your case is that we do have a guide for configuring Gruntwork Pipelines. You'd be looking to deploy the ECS Deploy Runner stack, which has several components. We also have documentation for this stack.

Then, regardless of whether you're currently using CircleCI, Jenkins or GitLab, you can see the configurations for these CI tools within the for-production infra-live repo, at the root of that repo. The relevant files will be the Jenkinsfile, gitlab-ci.yml, and the .circleci folder.

We also have deployment walkthroughs for how to use Pipelines after you've installed it:

• Two minute overview of Pipelines
• Apply walkthrough
• Destroy walkthrough

[zackproser]

Credit for this solution goes to @rhoboat

[rhoboat]

🤗