Terragrunt repository/directory layout to implement least access privilege

[blueelephants]

I am currently working on a Terragrunt GitHub repository/directory layout setup for deploying the "core cloud infrastructure setup" of an corporate Azure environment, which should consists of multiple components:

• networking hubs in multiple regions (for hub and spoke networking)
• central logging stack in multiple regions
• (and a couple of more corporate components ..)

I need to have two fully isolated environments: prd and dev (changes to these environments need to be strictly isolated; state files should be on different storage accounts too, etc.)

For IT security requirements, I need to implement the least access privilege concept, which also needs to be applied for running automation code.
That means, instead of having one single Azure Service Principal (i.e. automation user) which is configured as superuser (so-called Owner in Azure) on the entire cloud environment, I need to have muliple ones - e.g.

• network-hub-automation-prd
• network-hub-automation-dev
• central-logging-automation-prd
• central-logging-automation-dev

which are configured with the minimum permissions necessary on Azure (e.g. network-hub-automation-prd should only be able to deploy the network related resources in the configured prd subscriptions).

In addition, people doing dev work shouldn't be able to run prd automation (unless they are also member of the prd team) - segration of duties.

This is all to reduce the blast radius if dev/prd mistakes happen and/or any credentials for automation users get leaked, etc.

Deployment of these different components of the core infrastructure should then be done via individual GitHub actions.

I did lots of research on the Internet to find out the most suitable (mono/poly) repository and directory layout, but I am still not sure which way to go forward.

Currently I can think of these layouts:

(A) One mono repository

Repository: live-infrastructure

/
├── .github
│   ├── workflows
│   │   ├── dev
│   │   │   ├── north-europe
│   │   │   │   ├── central-logging
│   │   │   │   │   action.yaml
│   │   │   │   ├── networking-hub
│   │   │   │   │   action.yaml
│   │   │   ├── west-europe
│   │   │   │   ├── central-logging
│   │   │   │   │   action.yaml
│   │   │   │   ├── networking-hub
│   │   │   │   │   action.yaml
│   │   ├── prd
│   │   │   ├── north-europe
│   │   │   │   ├── central-logging
│   │   │   │   │   action.yaml
│   │   │   │   ├── networking─ hub
│   │   │   │   │   action.yaml
│   │   │   ├── west-europe
│   │   │   │   ├── central-logging
│   │   │   │   │   action.yaml
│   │   │   │   ├── networking-hub
│   │   │   │   │   action.yaml
├── live
│   terragrunt.hcl
│   ├── dev
│   │   env.hcl
│   │   ├── north-europe
│   │   │   region.hcl
│   │   │   ├── central-logging
│   │   │   │   terragrunt.hcl
│   │   │   ├── networking-hub
│   │   │   │   terragrunt.hcl
│   │   ├── west-europe
│   │   │   region.hcl
│   │   │   ├── central-logging
│   │   │   │   terragrunt.hcl
│   │   │   ├── networking-hub
│   │   │   │   terragrunt.hcl
│   ├── prd
│   │   env.hcl
│   │   ├── north-europe
│   │   │   region.hcl
│   │   │   ├── central-logging
│   │   │   │   terragrunt.hcl
│   │   │   ├── networking-hub
│   │   │   │   terragrunt.hcl
│   │   ├── west-europe
│   │   │   region.hcl
│   │   │   ├── central-logging
│   │   │   │   terragrunt.hcl
│   │   │   ├── networking-hub
│   │   │   │   terragrunt.hcl

• This is following the approach as laid out here: https://github.com/gruntwork-io/terragrunt-infrastructure-live-example.
• Dev/Prd credentials for the automation users (service principals) could be stored as GitHub repository secrets and be reused in the different GitHub action.yaml
• However, this would mean that people doing changes on dev would work in the same repository as people doing changes in prd.
• This might complicate "segration of duties" and isololation of dev/prd work as also all dev/prd people would be able to see/use all configured GitHub dev/prd repository secrets

(B) One repository per environment (poly-repo)

Repository: live-infrastructure-dev

/
├── .github
│   ├── workflows
│   │   ├── dev
│   │   │   ├── north-europe
│   │   │   │   ├── central-logging
│   │   │   │   │   action.yaml
│   │   │   │   ├── networking-hub
│   │   │   │   │   action.yaml
│   │   │   ├── west-europe
│   │   │   │   ├── central-logging
│   │   │   │   │   action.yaml
│   │   │   │   ├── networking-hub
│   │   │   │   │   action.yaml
├── live
│   terragrunt.hcl
│   ├── dev
│   │   env.hcl
│   │   ├── north-europe
│   │   │   region.hcl
│   │   │   ├── central-logging
│   │   │   │   terragrunt.hcl
│   │   │   ├── networking-hub
│   │   │   │   terragrunt.hcl
│   │   ├── west-europe
│   │   │   region.hcl
│   │   │   ├── central-logging
│   │   │   │   terragrunt.hcl
│   │   │   ├── networking-hub
│   │   │   │   terragrunt.hcl

Repository: live-infrastructure-prd

/
├── .github
│   ├── workflows
│   │   ├── prd
│   │   │   ├── north-europe
│   │   │   │   ├── central-logging
│   │   │   │   │   action.yaml
│   │   │   │   ├── networking-hub
│   │   │   │   │   action.yaml
│   │   │   ├── west-europe
│   │   │   │   ├── central-logging
│   │   │   │   │   action.yaml
│   │   │   │   ├── networking-hub
│   │   │   │   │   action.yaml
├── live
│   terragrunt.hcl
│   ├── prd
│   │   env.hcl
│   │   ├── north-europe
│   │   │   region.hcl
│   │   │   ├── central-logging
│   │   │   │   terragrunt.hcl
│   │   │   ├── networking-hub
│   │   │   │   terragrunt.hcl
│   │   ├── west-europe
│   │   │   region.hcl
│   │   │   ├── central-logging
│   │   │   │   terragrunt.hcl
│   │   │   ├── networking-hub
│   │   │   │   terragrunt.hcl

• Dev credentials for the automation users (service principals) could be stored as GitHub repository secrets in live-infrastructure-dev repository only
• Prd credentials for the automation users (service principals) could be stored as GitHub repository secrets in live-infrastructure-prd repository only
• This would give much better isolation between the dev/prd teams, as they have only access to one of the repositories and the configured GitHub respository secrets
• However, somebody could still mistakenly configure a GitHub action to run e.g. the network-hub automation with the GitHub repository secrets for the central-logging automation

(C) One repository per component per environment (poly-repo)

Repository: live-infrastructure-central-loging-dev

/
├── .github
│   ├── workflows
│   │   ├── dev
│   │   │   ├── north-europe
│   │   │   │   ├── central-logging
│   │   │   │   │   action.yaml
│   │   │   ├── west-europe
│   │   │   │   ├── central-logging
│   │   │   │   │   action.yaml
├── live
│   terragrunt.hcl
│   ├── dev
│   │   env.hcl
│   │   ├── north-europe
│   │   │   region.hcl
│   │   │   ├── central-logging
│   │   │   │   terragrunt.hcl
│   │   ├── west-europe
│   │   │   region.hcl
│   │   │   ├── central-logging
│   │   │   │   terragrunt.hcl

Repository: live-infrastructure-networking-hub-dev

/
├── .github
│   ├── workflows
│   │   ├── dev
│   │   │   ├── north-europe
│   │   │   │   ├── networking-hub
│   │   │   │   │   action.yaml
│   │   │   ├── west-europe
│   │   │   │   ├── networking-hub
│   │   │   │   │   action.yaml
├── live
│   terragrunt.hcl
│   ├── dev
│   │   env.hcl
│   │   ├── north-europe
│   │   │   region.hcl
│   │   │   ├── networking-hub
│   │   │   │   terragrunt.hcl
│   │   ├── west-europe
│   │   │   region.hcl
│   │   │   ├── networking-hub
│   │   │   │   terragrunt.hcl

Repository: live-infrastructure-central-loging-prd

/
├── .github
│   ├── workflows
│   │   ├── prd
│   │   │   ├── north-europe
│   │   │   │   ├── central-logging
│   │   │   │   │   action.yaml
│   │   │   ├── west-europe
│   │   │   │   ├── central-logging
│   │   │   │   │   action.yaml
├── live
│   terragrunt.hcl
│   ├── prd
│   │   env.hcl
│   │   ├── north-europe
│   │   │   region.hcl
│   │   │   ├── central-logging
│   │   │   │   terragrunt.hcl
│   │   ├── west-europe
│   │   │   region.hcl
│   │   │   ├── central-logging
│   │   │   │   terragrunt.hcl

Repository: live-infrastructure-networking-hub-prd

/
├── .github
│   ├── workflows
│   │   ├── prd
│   │   │   ├── north-europe
│   │   │   │   ├── networking-hub
│   │   │   │   │   action.yaml
│   │   │   ├── west-europe
│   │   │   │   ├── networking-hub
│   │   │   │   │   action.yaml
├── live
│   terragrunt.hcl
│   ├── prd
│   │   env.hcl
│   │   ├── north-europe
│   │   │   region.hcl
│   │   │   ├── networking-hub
│   │   │   │   terragrunt.hcl
│   │   ├── west-europe
│   │   │   region.hcl
│   │   │   ├── networking-hub
│   │   │   │   terragrunt.hcl

• Small repositories doing just the necessary work are handy seem to offer the best isolation
• However, this all comes at the cost of (too??) many repositories, which introduce maintenance costs

I would be happy if you can share your thoughts and experience with me and any good/best practices.

---

Tracked in ticket #110107

[denis256]

Hi,
in projects where I helped, people mostly used the mono repository approach(A), in some cases approach with one repository per environment(B)

[brikis98]

This is more of a question of your deployment workflow than of Terragrunt. Even if you weren't using Terragrunt, and solely using Terraform, you'd still have to resolve the question of "how do I decide which parts of my infrastructure code can be updated by which devs?"

There are many different ways to answer this question. Here are just a few:

1. Use multiple repos, and to only grant certain devs access to certain repos. This is the idea you mention in your post above, and it can work, but please be aware of the trade offs between a monorepo vs polyrepos for managing your live infrastructure. Also, note that depending on how granular your access privileges need to be, creating separate repos for them can get quite messy. If it's really just "prod vs non-prod," then two repos will do the trick. But if it gets more complicated—e.g., one team should be able to update A and B but not C or D, whereas another team can update A and C, but not B or D—then shuffling things around many different repos may get messy.
2. Code review. Require that (a) all changes to your repos to be submitted via pull requests and (b) that pull requests must be reviewed by at least 1 non-author before they can be merged (see GitHub protected branches. If you do that, then the reviewers of the PR can see who submitted it and if that person shouldn't be allowed to update that part of the code, you can reject the PR.
3. CI checks. Closely related to the previous item, instead of (or in addition to) human reviewers, you could have a CI server (e.g., GitHub Actions, CircleCI, Jenkins) that runs against every PR, looks at each commit, sees who made the changes, if they are allowed to change those files (i.e., those part of your infrastructure), and if not, fail the build, and blocking merge of the PR (you can enable this as part of protected branches). You should also enforce commit signing so commits can't be spoofed.
4. Deployment checks. Closely related to the previous item, you could have your deployment system enforce access controls. The assumption here is that you deploy changes with some automated system (e.g., GitHub Actions, CircleCI, etc) and that system, before doing a deployment on some committed code, scans the code to see who made the changes, if they are allowed to change those files (i.e., those part of your infrastructure), and if not, don't do the deployment. Again, enforced commit signing is a good idea here.

I'd recommend using items (2)-(4) in most cases, as they give you a bunch of nice benefits, including enforcing access controls as per this discussion. Note that all of these items work with monorepos or polyrepos too.

[blueelephants]

Thanks a lot for your feedback - this is very valuable food for thought for me.
I will explore/try out these options in more detail to come up with a solution to my requirements.