There is a backward incompatibility in terraform-aws-service-catalog for eks-cluster v0.63.4

[yorinasub17]

I am upgrading my eks-cluster module call from v0.58.5 to v0.63.4 with self-managed workers and the update is causing terraform to recreate the ASGs associated with my cluster. v0.63.4 should have been marked as backward incompatible.

What can I do to avoid the worker node recreation?

[gruntwork-discussions]

You must label this issue with at least one of the following labels before it can be answered:

• r:terraform-aws-service-catalog
• r:terraform-aws-vpc
• r:terraform-aws-architecture-catalog
• r:terraform-aws-asg
• r:terraform-aws-beanstalk
• r:terraform-aws-cache
• r:terraform-aws-ci
• r:terraform-aws-ci-pipeline-example
• r:terraform-aws-cis-service-catalog
• r:terraform-aws-data-storage
• r:terraform-aws-ecs
• r:terraform-aws-eks
• r:terraform-aws-elk
• r:terraform-aws-kafka
• r:terraform-aws-lambda
• r:terraform-aws-load-balancer
• r:terraform-aws-messaging
• r:terraform-aws-mongodb
• r:terraform-aws-monitoring
• r:terraform-aws-openvpn
• r:terraform-aws-sam
• r:terraform-aws-security
• r:terraform-aws-server
• r:terraform-aws-static-assets
• r:terraform-aws-zookeeper
• r:terragrunt
• r:terratest
• r:repo-copier

Community Guidelines
Getting help
Gruntwork documentation

[jeffreymlewis]

I went ahead and upgraded from v0.58.5 to v0.63.3. There were a new launch template changes (user-data updates) but no big deal.

When bumping from v0.63.3 to v0.63.4 I get a destroy/create cycle on all my ASGs, launch templates, etc, as shown in this example. Note the # forces replacement tag on the name_prefix parameter.

  # module.eks_workers["enabled"].module.self_managed_workers.aws_autoscaling_group.eks_worker["2xlarge-subnet-06260427f486a8585"] must be replaced
-/+ resource "aws_autoscaling_group" "eks_worker" {
      ~ arn                       = "arn:aws:autoscaling:us-east-1:968332900958:autoScalingGroup:16241724-4a14-4189-8ad5-f5b7ce95e468:autoScalingGroupName/dev00-ue1-app-eks-app-01-2xlarge-subnet-06260427f486a8585-2022011222453498950000000d" -> (known after apply)
      ~ availability_zones        = [
          - "us-east-1a",
        ] -> (known after apply)
      - capacity_rebalance        = false -> null
      ~ default_cooldown          = 300 -> (known after apply)
      ~ desired_capacity          = 0 -> (known after apply)
      - enabled_metrics           = [] -> null
        force_delete              = false
        force_delete_warm_pool    = false
        health_check_grace_period = 300
      ~ health_check_type         = "EC2" -> (known after apply)
      ~ id                        = "dev00-ue1-app-eks-app-01-2xlarge-subnet-06260427f486a8585-2022011222453498950000000d" -> (known after apply)
      - load_balancers            = [] -> null
      - max_instance_lifetime     = 0 -> null
        max_size                  = 40
        metrics_granularity       = "1Minute"
        min_size                  = 0
      ~ name                      = "dev00-ue1-app-eks-app-01-2xlarge-subnet-06260427f486a8585-2022011222453498950000000d" -> (known after apply)
      ~ name_prefix               = "dev00-ue1-app-eks-app-01-2xlarge-subnet-06260427f486a8585-" -> (known after apply) # forces replacement
        protect_from_scale_in     = false
      ~ service_linked_role_arn   = "arn:aws:iam::968332900958:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling" -> (known after apply)
      - suspended_processes       = [] -> null
      - target_group_arns         = [] -> null
      - termination_policies      = [] -> null
        vpc_zone_identifier       = [
            "subnet-06260427f486a8585",
        ]
        wait_for_capacity_timeout = "10m"

      ~ mixed_instances_policy {
          ~ instances_distribution {
              ~ on_demand_allocation_strategy            = "prioritized" -> (known after apply)
              ~ on_demand_base_capacity                  = 0 -> (known after apply)
              ~ on_demand_percentage_above_base_capacity = 0 -> (known after apply)
                spot_allocation_strategy                 = "capacity-optimized"
                spot_instance_pools                      = 0
            }

          ~ launch_template {
              ~ launch_template_specification {
                  ~ launch_template_id   = "lt-04ccd97a3494a441c" -> (known after apply)
                  ~ launch_template_name = "dev00-ue1-app-eks-app-01-2xlarge-subnet-06260427f486a8585-20220112224347187700000006" -> (known after apply)
                  ~ version              = "2" -> (known after apply)
                }
                override {
                    instance_type     = "m6i.2xlarge"
                    weighted_capacity = "1"
                }
                override {
                    instance_type     = "m6a.2xlarge"
                    weighted_capacity = "1"
                }
                override {
                    instance_type     = "m5.2xlarge"
                    weighted_capacity = "1"
                }
                override {
                    instance_type     = "m5a.2xlarge"
                    weighted_capacity = "1"
                }
                override {
                    instance_type     = "m4.2xlarge"
                    weighted_capacity = "1"
                }
            }
        }

        tag {
            key                 = "Name"
            propagate_at_launch = true
            value               = "dev00-ue1-app-eks-app-01"
        }
      + tag {
          + key                 = "Name"
          + propagate_at_launch = true
          + value               = (known after apply)
        }
      - tag {
          - key                 = "k8s.io/cluster-autoscaler/dev00-ue1-app-eks-app-01" -> null
          - propagate_at_launch = true -> null
          - value               = "true" -> null
        }
        tag {
            key                 = "k8s.io/cluster-autoscaler/enabled"
            propagate_at_launch = true
            value               = "true"
        }
      - tag {
          - key                 = "kubernetes.io/cluster/dev00-ue1-app-eks-app-01" -> null
          - propagate_at_launch = true -> null
          - value               = "owned" -> null
        }
      + tag {
          + key                 = (known after apply)
          + propagate_at_launch = true
          + value               = "owned"
        }
      + tag {
          + key                 = (known after apply)
          + propagate_at_launch = true
          + value               = "true"
        }
    }

[jeffreymlewis]

Getting a destroy/create cycle on managed node groups as well. Not sure what's up with all the # forces replacement. The new parameter node_group_name_prefix might be the cause of this one?

  # module.eks_workers["enabled"].module.managed_node_groups.aws_eks_node_group.eks_worker["managed-spot-subnet-05e6cbfeea7f38404"] must be replaced
-/+ resource "aws_eks_node_group" "eks_worker" {
      ~ ami_type               = "CUSTOM" -> (known after apply) # forces replacement
      ~ arn                    = "arn:aws:eks:us-east-1:968332900958:nodegroup/dev00-ue1-app-eks-app-01/dev00-ue1-app-eks-app-01-managed-spot-subnet-05e6cbfeea7f38404/d0bf2d52-7512-c048-4162-fcbeb156392e" -> (known after apply)
      ~ capacity_type          = "SPOT" -> (known after apply) # forces replacement
      ~ cluster_name           = "dev00-ue1-app-eks-app-01" -> (known after apply) # forces replacement
      ~ disk_size              = 0 -> (known after apply) # forces replacement
      ~ id                     = "dev00-ue1-app-eks-app-01:dev00-ue1-app-eks-app-01-managed-spot-subnet-05e6cbfeea7f38404" -> (known after apply)
      ~ instance_types         = [
          - "m6i.2xlarge",
          - "m6a.2xlarge",
          - "m5.2xlarge",
          - "m5a.2xlarge",
        ] -> (known after apply) # forces replacement
      ~ labels                 = {} -> (known after apply)
      ~ node_group_name        = "dev00-ue1-app-eks-app-01-managed-spot-subnet-05e6cbfeea7f38404" -> (known after apply) # forces replacement
      + node_group_name_prefix = (known after apply)
        node_role_arn          = "arn:aws:iam::968332900958:role/dev00-ue1-app-eks-app-01-mng"
      ~ release_version        = "ami-0c80e8f97e9006c3e" -> (known after apply)
      ~ resources              = [
          - {
              - autoscaling_groups              = [
                  - {
                      - name = "eks-dev00-ue1-app-eks-app-01-managed-spot-subnet-05e6cbfeea7f38404-d0bf2d52-7512-c048-4162-fcbeb156392e"
                    },
                ]
              - remote_access_security_group_id = ""
            },
        ] -> (known after apply)
      ~ status                 = "ACTIVE" -> (known after apply)
      ~ subnet_ids             = [
          - "subnet-05e6cbfeea7f38404",
        ] -> (known after apply) # forces replacement
      ~ tags                   = {} -> (known after apply)
      ~ tags_all               = {} -> (known after apply)
      ~ version                = "1.21" -> (known after apply)

      ~ launch_template {
          ~ id      = "lt-0915f78d7008ff364" -> (known after apply) # forces replacement
          ~ name    = "dev00-ue1-app-eks-app-0120220114221102600800000004" -> (known after apply) # forces replacement
          ~ version = "1" -> (known after apply)
        }

      + remote_access { # forces replacement
          + ec2_ssh_key               = (known after apply) # forces replacement
          + source_security_group_ids = (known after apply) # forces replacement
        }

      ~ scaling_config {
          ~ desired_size = 1 -> (known after apply)
          ~ max_size     = 10 -> (known after apply)
          ~ min_size     = 1 -> (known after apply)
        }

      ~ update_config {
          ~ max_unavailable            = 1 -> (known after apply)
          ~ max_unavailable_percentage = 0 -> (known after apply)
        }
    }

[yorinasub17]

I have identified the root cause for this diff. It is caused by this PR, https://github.com/gruntwork-io/terraform-aws-service-catalog/pull/994, which introduced a new null_resource that is skewing the dependency chain.

I have confirmed that running the following should resolve this recreation:

terragrunt apply -target null_resource.depend_on_auth_merger # on v0.63.4. Make sure the plan looks minimal!

Once the null_resource is created, everything should be knowable at plan time and thus avoid all the recreations. This applies to both self managed and managed worker nodes.

[yorinasub17]

I have identified the root cause for this diff. It is caused by this PR, https://github.com/gruntwork-io/terraform-aws-service-catalog/pull/994, which introduced a new null_resource that is skewing the dependency chain.

I have confirmed that running the following should resolve this recreation:

terragrunt apply -target null_resource.depend_on_auth_merger # on v0.63.4. Make sure the plan looks minimal!

Once the null_resource is created, everything should be knowable at plan time and thus avoid all the recreations. This applies to both self managed and managed worker nodes.