Failed to query available provider packages and connect to RDS on terraform cloud

[fabiobsantosprogrow]

When I use the provider example I found this error:

The configuration used for this test:

terraform {
  required_providers {
    aws-ssm-tunnels = {
      source = "ComplyCo/aws-ssm-tunnels"
      version = "0.2.1"
    }
  }

provider "awsssmtunnels" {
  region     = var.aws_region
  access_key = var.aws_access_key
  secret_key = var.aws_secret_key
}

When I change to:

provider "aws-ssm-tunnels" {
  region     = var.aws_region
  access_key = var.aws_access_key
  secret_key = var.aws_secret_key
}

Result:

When I try to create awsssmtunnels_remote_tunnel using this example the same error happens again.
Configuration:

resource "awsssmtunnels_remote_tunnel" "rds" {
  refresh_id  = "one" // Anything string can go here as this resource will always find a diff on this
  target      = "i-000000000000000000"
  remote_host = "*.rds.amazonaws.com"
  remote_port = 3306 
  local_port  = 17638
  region      = var.aws_region
}

What is the problem with this configuration?

[sdemjanenko]

Can you try changing

terraform {
  required_providers {
    aws-ssm-tunnels = {
      source = "ComplyCo/aws-ssm-tunnels"
      version = "0.2.1"
    }
  }
}

to

terraform {
  required_providers {
    awsssmtunnels = {
      source = "ComplyCo/aws-ssm-tunnels"
      version = "0.2.1"
    }
  }
}

Terraform's provider naming is a bit annoying when it comes to hyphens.

[fabiobsantosprogrow]

After following your advice I can run and apply this on terraform cloud environment but i get this error:

The tunnel was created but we cannot perform connection to the mysql cluster using the mysql provider:

Questions:
This providers runs on terraform cloud?
How to throubleshoot the problem?

Kind regards

[sdemjanenko]

@fabiobsantosprogrow glad that moved you a step further. The issue you are hitting might be happening if Terraform thinks it is done with the awsssmtunnels provider too early. This provider does run on TF Cloud.

Do you have this in your TF code

resource "awsssmtunnels_remote_tunnel" "rds" {
...
}

// NOTE: The import is needed for the first plan, otherwise TF will hold off on the create until the Apply phase.
// The import allows it to run in the very first plan.
import {
  id = "<target>|<remote host>|<remote port>|<local port>|127.0.0.1|<region>"
  to = awsssmtunnels_remote_tunnel.rds
}

// Keepalive prevents TF from prematurely sending a shutdown command to the Provider process (which would kill the tunnel)
data "awsssmtunnels_keepalive" "rds" {
  depends_on = [
    awsssmtunnels_remote_tunnel.rds,
    ... every MySQL resource
  ]
}

// Add depends_on so that TF doesn't do anything with these resources until the tunnel is up
resource "mysql_...." "..." {
  ...
  depends_on = [awsssmtunnels_remote_tunnel.rds]
}

Let me know if that helps, or if you continue to run into issues.

[fabiobsantosprogrow]

I try your suggestion and the connection problem keep going:

I also check the SSM configuration using the command line to open a port to the desired database and it works:
aws --profile profileName ssm start-session --target i-************** --document-name AWS-StartPortForwardingSessionToRemoteHost --parameters '{"host":["*.eu-west-1.rds.amazonaws.com"],"portNumber":["3306"], "localPortNumber":["43310"]}'
Any suggestion?

[sdemjanenko]

@fabiobsantosprogrow do you have something like this in your TF code?

data "awsssmtunnels_keepalive" "rds" {
  depends_on = [
    awsssmtunnels_remote_tunnel.rds,
    mysql_user.jdoe, // NOTE: This resource is registered specifically
  ]
}

Would it be possible for you to share a small TF code example that exhibits this problem (with stubbed names/settings)? That would help me conceptualize how you have it set up.

Also are you opening multiple SSM tunnels?

[sdemjanenko]

I did some debugging with the multi-tunnel scenario and I found that for reliability, one should have a provider per tunnel (due to the AWS session-manager-plugin issuing os.Exit which kills the provider even though other tunnels are open).

provider "awsssmtunnels" {
  alias               = "rds" // NOTE: The alias here
  region              = "us-east-1"
  shared_config_files = [var.tfc_aws_dynamic_credentials.default.shared_config_file]
}

resource "awsssmtunnels_remote_tunnel" "rds" {
  provider    = awsssmtunnels.rds // NOTE: The provider alias reference here
  refresh_id  = "one"
  target      = var.bastion_instance_id
  remote_host = var.endpoint
  remote_port = 5432
  region      = "us-east-1"
  local_port  = 16534
}

import {
  id = "${var.bastion_instance_id}|${var.endpoint}|5432|16534|127.0.0.1|us-east-1"
  to = awsssmtunnels_remote_tunnel.rds
}

data "awsssmtunnels_keepalive" "rds" {
  provider = awsssmtunnels.rds // The provider alias reference here
  depends_on = [
    ...
    awsssmtunnels_remote_tunnel.rds,
  ]
}

You can then have another provider like

provider "awsssmtunnels" {
  alias               = "eks" // NOTE: The alias here
  region              = "us-east-1"
  shared_config_files = [var.tfc_aws_dynamic_credentials.default.shared_config_file]
}

Running multiple aliased providers this way will spin up separate processes, one per provider and that isolation seems to help it run more reliably.

[fabiobsantosprogrow]

Great work!
Using version 3.0.0 I with single tunnel I can create an user using mysql provider but when I try to run second time the terraform hangs on this step and fails:

I have some problems with this piece of code:
import { id = "<target>|<remote host>|<remote port>|<local port>|127.0.0.1|<region>" to = awsssmtunnels_remote_tunnel.rds }
Until I remove this piece of code I have this message error: "remotePort must be set"

Update:

setting the mysql provider endpoint harcoded allows to run again terraform. Now I'm struggling with delete the user that I previously created. Because of data "awsssmtunnels_keepalive" with user as dependency when I try to delete the user by removing on the terraform file the resource "mysql_user" "jdoe" and next setting count to value zero the resource is hanging on destroy:

[sdemjanenko]

@fabiobsantosprogrow you raise a good question about deletion of a resource (such as a user). I'm not sure that this provider can easily support that. I'll have to do some experimenting. I did want to highlight this issue in Terraform (hashicorp/terraform#8367) which if implemented could make this provider simpler to configure and make resource deletion support work properly.