Shared "Cache" of Terraform Providers

[dhoffman-vertex]

Greetings!

I am trying to accomplish something, that I know must be possible and I am just missing it in the documentation.

I am trying to make it so that I can have a shared cache of Terraform Providers that is used by ALL modules within my Terragrunt Project.

I have over 475 modules (all AWS based), and it doesn't seem sane to me that I would need to download the AWS provider 475 times, when once should be all that is required. I have tried using the terragrunt-cache mechanism, but as noted I then have to limit parrallelism to 1.

Is there a better way?

---

Tracked in ticket #110845

[yhakbar]

Hey @dhoffman-vertex !

There is a long, in-depth discussion about this topic here.

The TLDR is that you can use standard provider plugin caching, but you can experience errors when running multiple Terragrunt operations concurrently while using it, as Terraform has no locking mechanism that makes it safe to access from multiple concurrent sources.

The simplest way to mitigate that drawback is to init with a parallelism of 1, then run the operation you want without limiting parallelism. That will populate the shared provider cache in a sequential fashion, then the plan, apply, etc can be run in parallel.

[brikis98]

Note that we're also working on a better way to cache providers, even with run-all / concurrency, in gruntwork-io/terragrunt#3001.

[dhoffman-vertex]

Thank you @yhakbar and @brikis98

While we wait for the ultimate solution in gruntwork-io/terragrunt#3001, I am posting the solution I ultimately came to given the information @yhakbar provided in hopes that if others come here looking for an answer, they will be able to leverage what I have learned over the last 24 hours.

1. I opted for a non-standard localation for my terraformrc file, the location of this is specified with the TF_CLI_CONFIG_FILE config variable.

plugin_cache_dir = "/tmp/terraform-providers"
plugin_cache_may_break_dependency_lock_file = true

provider_installation {
  filesystem_mirror {
    path    = "/tmp/terraform-providers"
    include = ["*/*"]
  }
  direct {
    exclude = ["*/*"]
  }
}

2. I found I had to "prepopulate" my mirror with the providers. In order to do so I had to tell TG to ignore the custom .terraformrc file.

I creates a simple providers.tf file

provider "aws" {}

Then, by unsetting the value of the TS_CLI_CONFIG_FILE in my command line, I "mirrored" the latest version of the providers

TF_CLI_CONFIG_FILE= terraform providers mirror /tmp/terraform-providers/

3. As noted by @yhakbar, I did a terragrunt init with parallelism setting of 1

terragrunt run-all init --terragrunt-parallelism 1

4. Then I was able to apply with a higher parallelism value. I did have to instruct it not to re-init, unsure why, but it would still try to init otherwise...

NOTE: I found I had to limit the parallelism to 10 in my case, but I believe that to be more of the circumstances of my environment (ephemeral GitHub Action Runners in Kubernetes) than that of Terragrunt.

terragrunt run-all apply --terragrunt-no-auto-init --terragrunt-parallelism 10

NOTE: I did note that there were many warnings when running the apply with the no-auto-init that it had detected a need to reinit and that further attempts could fail, but it seemed to work fine. Ultimately I opted to add a --terragrunt-log-level error flag to squelch the warnings.