ARPA audit report task fixes

[TylerHendrickson]

Description

This is a maintenance PR that tweaks a bit of log output and fixes a few issues found while testing the new task-based solution for ARPA audit report generation:

1. ECS tasks were failing to start because the service networking configuration was lacking a security group attachment that would allow ingress to the EFS volume mount. This is solved by ensuring the correct SG is configured on the service.
2. The new processSQSMessageRequest() function was accessing a nonexistent attribute on parsed SQS message bodies. This was more or less a typo, which this PR removes.
3. The refactor commit a492c06 in Add task queue for ARPA audit reports #1992 missed an instance where a tenantId value needs to be passed in a function call in order to avoid an indirect call to useTenant() (which fails outside of request-handling contexts).

[codeclimate]

Code Climate has analyzed commit f7a9138 and detected 0 issues on this pull request.

The test coverage on the diff in this pull request is 15.3% (50% is the threshold).

This pull request will bring the total coverage in the repository to 55.8% (0.0% change).

View more on Code Climate.

[github-actions]

Report for project: terraform

Terraform Initialization ⚙️success

Terraform Validation 🤖success

Validation Output

Success! The configuration is valid.

Terraform Plan 📖success

Show Plan

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the
last "terraform apply" which may have affected this plan:

  # module.api.aws_ecs_task_definition.default[0] has changed
  ~ resource "aws_ecs_task_definition" "default" {
        id                       = "gost-staging-api"
+       tags                     = {}
        # (13 unchanged attributes hidden)

+       volume {
+           name = "data"

+           efs_volume_configuration {
+               file_system_id          = "fs-08f95063c1cdbe191"
+               root_directory          = "/"
+               transit_encryption      = "ENABLED"
+               transit_encryption_port = 0

+               authorization_config {
+                   access_point_id = "fsap-03bc0296928aade4f"
                }
            }
        }
-       volume {
-           name = "data" -> null

-           efs_volume_configuration {
-               file_system_id     = "fs-08f95063c1cdbe191" -> null
-               root_directory     = "/" -> null
-               transit_encryption = "ENABLED" -> null

-               authorization_config {
-                   access_point_id = "fsap-03bc0296928aade4f" -> null
                }
            }
        }

        # (1 unchanged block hidden)
    }

  # module.api.aws_iam_role.execution[0] has changed
  ~ resource "aws_iam_role" "execution" {
        id                    = "gost-staging-api-ECSTaskExecution-20230217010414321500000009"
        name                  = "gost-staging-api-ECSTaskExecution-20230217010414321500000009"
      ~ role_last_used        = [
          ~ {
              ~ last_used_date = "2023-09-29T13:02:17Z" -> "2023-09-29T22:18:03Z"
                # (1 unchanged element hidden)
            },
        ]
        tags                  = {}
        # (11 unchanged attributes hidden)

        # (2 unchanged blocks hidden)
    }

  # module.api.aws_iam_role.task[0] has changed
  ~ resource "aws_iam_role" "task" {
        id                    = "gost-staging-api-ECSTask-2023021701041477300000000a"
        name                  = "gost-staging-api-ECSTask-2023021701041477300000000a"
      ~ role_last_used        = [
          ~ {
              ~ last_used_date = "2023-09-29T13:02:17Z" -> "2023-09-29T20:16:21Z"
                # (1 unchanged element hidden)
            },
        ]
        tags                  = {}
        # (11 unchanged attributes hidden)

+       inline_policy {
+           name   = "send-arpa-audit-report-requests20230929145125577500000001"
+           policy = jsonencode(
                {
+                   Statement = [
+                       {
+                           Action   = "sqs:SendMessage"
+                           Effect   = "Allow"
+                           Resource = "arn:aws:sqs:us-west-2:357150818708:gost-staging-arpa_audit_report-20230929024256614400000005"
+                           Sid      = "AllowPublishToQueue"
                        },
                    ]
+                   Version   = "2012-10-17"
                }
            )
        }

        # (4 unchanged blocks hidden)
    }


Unless you have made equivalent changes to your configuration, or ignored the
relevant attributes using ignore_changes, the following plan may include
actions to undo or respond to these changes.

─────────────────────────────────────────────────────────────────────────────

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
+   create
  ~ update in-place
-   destroy
-/+ destroy and then create replacement
+/- create replacement and then destroy

Terraform will perform the following actions:

  # module.api.aws_ecs_service.default[0] will be updated in-place
  ~ resource "aws_ecs_service" "default" {
        id                                 = "arn:aws:ecs:us-west-2:357150818708:service/gost-staging/gost-staging-api"
        name                               = "gost-staging-api"
        tags                               = {}
      ~ task_definition                    = "arn:aws:ecs:us-west-2:357150818708:task-definition/gost-staging-api:33" -> (known after apply)
        # (15 unchanged attributes hidden)

      ~ network_configuration {
          ~ security_groups  = [
-               "sg-04836a204184dee25",
-               "sg-083ce3fb8d4e66859",
-               "sg-0b8d8e7a05714bd3f",
            ] -> (known after apply)
            # (2 unchanged attributes hidden)
        }

        # (3 unchanged blocks hidden)
    }

  # module.api.aws_ecs_task_definition.default[0] must be replaced
+/- resource "aws_ecs_task_definition" "default" {
      ~ arn                      = "arn:aws:ecs:us-west-2:357150818708:task-definition/gost-staging-api:33" -> (known after apply)
      ~ arn_without_revision     = "arn:aws:ecs:us-west-2:357150818708:task-definition/gost-staging-api" -> (known after apply)
      ~ container_definitions    = jsonencode(
          ~ [ # forces replacement
              ~ {
                  ~ dockerLabels           = {
                      ~ "com.datadoghq.tags.version" = "1e0c52a4041d6e2209a05a33b82148ccc859ea12" -> "8e06f67b29cf90cde5011c208ce59e4616a4a288"
                        # (2 unchanged elements hidden)
                    }
                  ~ environment            = [
                        # (6 unchanged elements hidden)
                        {
                            name  = "DD_SERVICE"
                            value = "gost"
                        },
                      ~ {
                            name  = "DD_VERSION"
                          ~ value = "1e0c52a4041d6e2209a05a33b82148ccc859ea12" -> "8e06f67b29cf90cde5011c208ce59e4616a4a288"
                        },
                        {
                            name  = "ENABLE_GRANTS_DIGEST"
                            value = "false"
                        },
                        # (9 unchanged elements hidden)
                    ]
                    name                   = "api"
-                   volumesFrom            = [] -> null
                    # (10 unchanged elements hidden)
                } # forces replacement,
              ~ {
                  ~ dockerLabels           = {
                      ~ "com.datadoghq.tags.version" = "1e0c52a4041d6e2209a05a33b82148ccc859ea12" -> "8e06f67b29cf90cde5011c208ce59e4616a4a288"
                        # (2 unchanged elements hidden)
                    }
                  ~ environment            = [
                        # (3 unchanged elements hidden)
                        {
                            name  = "DD_SERVICE"
                            value = "gost"
                        },
                      ~ {
                            name  = "DD_VERSION"
                          ~ value = "1e0c52a4041d6e2209a05a33b82148ccc859ea12" -> "8e06f67b29cf90cde5011c208ce59e4616a4a288"
                        },
                        {
                            name  = "ECS_FARGATE"
                            value = "true"
                        },
                    ]
-                   mountPoints            = [] -> null
                    name                   = "datadog"
-                   portMappings           = [] -> null
-                   volumesFrom            = [] -> null
                    # (6 unchanged elements hidden)
                } # forces replacement,
            ]
        )
      ~ id                       = "gost-staging-api" -> (known after apply)
      ~ revision                 = 33 -> (known after apply)
-       tags                     = {} -> null
        # (9 unchanged attributes hidden)

-       volume {
-           name = "data" -> null

-           efs_volume_configuration {
-               file_system_id          = "fs-08f95063c1cdbe191" -> null
-               root_directory          = "/" -> null
-               transit_encryption      = "ENABLED" -> null
-               transit_encryption_port = 0 -> null

-               authorization_config {
-                   access_point_id = "fsap-03bc0296928aade4f" -> null
                }
            }
        }
+       volume {
+           name = "data"

+           efs_volume_configuration {
+               file_system_id     = "fs-08f95063c1cdbe191"
+               root_directory     = "/"
+               transit_encryption = "ENABLED"

+               authorization_config {
+                   access_point_id = "fsap-03bc0296928aade4f"
                }
            }
        }

        # (1 unchanged block hidden)
    }

  # module.arpa_audit_report.aws_ecs_service.default will be updated in-place
  ~ resource "aws_ecs_service" "default" {
        id                                 = "arn:aws:ecs:us-west-2:357150818708:service/gost-staging/gost-staging-arpa_audit_report"
        name                               = "gost-staging-arpa_audit_report"
        tags                               = {}
      ~ task_definition                    = "arn:aws:ecs:us-west-2:357150818708:task-definition/gost-staging-arpa_audit_report:2" -> (known after apply)
        # (15 unchanged attributes hidden)

      ~ network_configuration {
          ~ security_groups  = [
-               "sg-09b90196ed40630de",
            ] -> (known after apply)
            # (2 unchanged attributes hidden)
        }

        # (2 unchanged blocks hidden)
    }

  # module.arpa_audit_report.aws_ecs_task_definition.consumer must be replaced
+/- resource "aws_ecs_task_definition" "consumer" {
      ~ arn                      = "arn:aws:ecs:us-west-2:357150818708:task-definition/gost-staging-arpa_audit_report:2" -> (known after apply)
      ~ arn_without_revision     = "arn:aws:ecs:us-west-2:357150818708:task-definition/gost-staging-arpa_audit_report" -> (known after apply)
      ~ container_definitions    = (sensitive value) # forces replacement
      ~ id                       = "gost-staging-arpa_audit_report" -> (known after apply)
-       ipc_mode                 = "" -> null
-       pid_mode                 = "" -> null
      ~ revision                 = 2 -> (known after apply)
-       tags                     = {} -> null
        # (9 unchanged attributes hidden)

-       volume {
-           name = "data" -> null

-           efs_volume_configuration {
-               file_system_id          = "fs-08f95063c1cdbe191" -> null
-               root_directory          = "/" -> null
-               transit_encryption      = "ENABLED" -> null
-               transit_encryption_port = 0 -> null

-               authorization_config {
-                   access_point_id = "fsap-03bc0296928aade4f" -> null
                }
            }
        }
+       volume {
+           name = "data"

+           efs_volume_configuration {
+               file_system_id     = "fs-08f95063c1cdbe191"
+               root_directory     = "/"
+               transit_encryption = "ENABLED"

+               authorization_config {
+                   access_point_id = "fsap-03bc0296928aade4f"
                }
            }
        }

        # (1 unchanged block hidden)
    }

  # module.arpa_audit_report_security_group.aws_security_group.cbd[0] will be created
+   resource "aws_security_group" "cbd" {
+       arn                    = (known after apply)
+       description            = "Managed by Terraform"
+       egress                 = (known after apply)
+       id                     = (known after apply)
+       ingress                = (known after apply)
+       name                   = (known after apply)
+       name_prefix            = (known after apply)
+       owner_id               = (known after apply)
+       revoke_rules_on_delete = false
+       tags                   = {
+           "Attributes" = "arpaauditreport"
+           "Name"       = "gost-staging-arpaauditreport"
+           "Namespace"  = "gost-staging"
        }
+       tags_all               = {
+           "Attributes" = "arpaauditreport"
+           "Name"       = "gost-staging-arpaauditreport"
+           "Namespace"  = "gost-staging"
+           "env"        = "staging"
+           "management" = "terraform"
+           "owner"      = "grants"
+           "repo"       = "usdr-gost"
+           "service"    = "gost"
+           "usage"      = "workload"
        }
+       vpc_id                 = (sensitive value)

+       timeouts {
+           create = "10m"
+           delete = "15m"
        }
    }

  # module.arpa_audit_report_security_group.aws_security_group_rule.keyed["_allow_all_egress_"] will be created
+   resource "aws_security_group_rule" "keyed" {
+       cidr_blocks              = [
+           "0.0.0.0/0",
        ]
+       description              = "Allow all egress"
+       from_port                = 0
+       id                       = (known after apply)
+       ipv6_cidr_blocks         = [
+           "::/0",
        ]
+       prefix_list_ids          = []
+       protocol                 = "-1"
+       security_group_id        = (known after apply)
+       security_group_rule_id   = (known after apply)
+       self                     = false
+       source_security_group_id = (known after apply)
+       to_port                  = 0
+       type                     = "egress"
    }

  # module.arpa_audit_report_security_group.null_resource.sync_rules_and_sg_lifecycles[0] will be created
+   resource "null_resource" "sync_rules_and_sg_lifecycles" {
+       id       = (known after apply)
+       triggers = {
+           "sg_ids" = (known after apply)
        }
    }

  # module.arpa_audit_report_security_group.random_id.rule_change_forces_new_security_group[0] will be created
+   resource "random_id" "rule_change_forces_new_security_group" {
+       b64_std     = (known after apply)
+       b64_url     = (known after apply)
+       byte_length = 3
+       dec         = (known after apply)
+       hex         = (known after apply)
+       id          = (known after apply)
+       keepers     = {
+           "rules" = jsonencode(
                {
+                   _allow_all_egress_ = {
+                       cidr_blocks              = [
+                           "0.0.0.0/0",
                        ]
+                       description              = "Allow all egress"
+                       from_port                = 0
+                       ipv6_cidr_blocks         = [
+                           "::/0",
                        ]
+                       key                      = "_allow_all_egress_"
+                       prefix_list_ids          = []
+                       protocol                 = "-1"
+                       security_groups          = []
+                       self                     = null
+                       source_security_group_id = null
+                       to_port                  = 0
+                       type                     = "egress"
                    }
                }
            )
        }
    }

  # module.arpa_audit_report_to_postgres_security_group.aws_security_group.cbd[0] will be destroyed
  # (because aws_security_group.cbd is not in configuration)
-   resource "aws_security_group" "cbd" {
-       arn                    = "arn:aws:ec2:us-west-2:357150818708:security-group/sg-04836a204184dee25" -> null
-       description            = "Managed by Terraform" -> null
-       egress                 = [
-           {
-               cidr_blocks      = [
-                   "0.0.0.0/0",
                ]
-               description      = "Allow all egress"
-               from_port        = 0
-               ipv6_cidr_blocks = [
-                   "::/0",
                ]
-               prefix_list_ids  = []
-               protocol         = "-1"
-               security_groups  = []
-               self             = false
-               to_port          = 0
            },
        ] -> null
-       id                     = "sg-04836a204184dee25" -> null
-       ingress                = [] -> null
-       name                   = "gost-staging-arpaauditreport-postgres--Bkcv-20230929020311423900000006" -> null
-       name_prefix            = "gost-staging-arpaauditreport-postgres--Bkcv-" -> null
-       owner_id               = "357150818708" -> null
-       revoke_rules_on_delete = false -> null
-       tags                   = {
-           "Attributes" = "arpaauditreport-postgres"
-           "Name"       = "gost-staging-arpaauditreport-postgres"
-           "Namespace"  = "gost-staging"
        } -> null
-       tags_all               = {
-           "Attributes" = "arpaauditreport-postgres"
-           "Name"       = "gost-staging-arpaauditreport-postgres"
-           "Namespace"  = "gost-staging"
-           "env"        = "staging"
-           "management" = "terraform"
-           "owner"      = "grants"
-           "repo"       = "usdr-gost"
-           "service"    = "gost"
-           "usage"      = "workload"
        } -> null
-       vpc_id                 = (sensitive value) -> null

-       timeouts {
-           create = "10m" -> null
-           delete = "15m" -> null
        }
    }

  # module.arpa_audit_report_to_postgres_security_group.aws_security_group_rule.keyed["_allow_all_egress_"] will be destroyed
  # (because aws_security_group_rule.keyed is not in configuration)
-   resource "aws_security_group_rule" "keyed" {
-       cidr_blocks       = [
-           "0.0.0.0/0",
        ] -> null
-       description       = "Allow all egress" -> null
-       from_port         = 0 -> null
-       id                = "sgrule-1650796685" -> null
-       ipv6_cidr_blocks  = [
-           "::/0",
        ] -> null
-       prefix_list_ids   = [] -> null
-       protocol          = "-1" -> null
-       security_group_id = "sg-04836a204184dee25" -> null
-       self              = false -> null
-       to_port           = 0 -> null
-       type              = "egress" -> null
    }

  # module.arpa_audit_report_to_postgres_security_group.null_resource.sync_rules_and_sg_lifecycles[0] will be destroyed
  # (because null_resource.sync_rules_and_sg_lifecycles is not in configuration)
-   resource "null_resource" "sync_rules_and_sg_lifecycles" {
-       id       = "718810493581168651" -> null
-       triggers = {
-           "sg_ids" = "sg-04836a204184dee25"
        } -> null
    }

  # module.arpa_audit_report_to_postgres_security_group.random_id.rule_change_forces_new_security_group[0] will be destroyed
  # (because random_id.rule_change_forces_new_security_group is not in configuration)
-   resource "random_id" "rule_change_forces_new_security_group" {
-       b64_std     = "Bkcv" -> null
-       b64_url     = "Bkcv" -> null
-       byte_length = 3 -> null
-       dec         = "411439" -> null
-       hex         = "06472f" -> null
-       id          = "Bkcv" -> null
-       keepers     = {
-           "rules" = jsonencode(
                {
-                   _allow_all_egress_ = {
-                       cidr_blocks              = [
-                           "0.0.0.0/0",
                        ]
-                       description              = "Allow all egress"
-                       from_port                = 0
-                       ipv6_cidr_blocks         = [
-                           "::/0",
                        ]
-                       key                      = "_allow_all_egress_"
-                       prefix_list_ids          = []
-                       protocol                 = "-1"
-                       security_groups          = []
-                       self                     = null
-                       source_security_group_id = null
-                       to_port                  = 0
-                       type                     = "egress"
                    }
                }
            )
        } -> null
    }

  # module.consume_grants.aws_ecs_service.default will be updated in-place
  ~ resource "aws_ecs_service" "default" {
        id                                 = "arn:aws:ecs:us-west-2:357150818708:service/gost-staging/gost-staging-consume_grants"
        name                               = "gost-staging-consume_grants"
        tags                               = {}
      ~ task_definition                    = "arn:aws:ecs:us-west-2:357150818708:task-definition/gost-staging-consume_grants:18" -> (known after apply)
        # (15 unchanged attributes hidden)

        # (3 unchanged blocks hidden)
    }

  # module.consume_grants.aws_ecs_task_definition.consume_grants must be replaced
+/- resource "aws_ecs_task_definition" "consume_grants" {
      ~ arn                      = "arn:aws:ecs:us-west-2:357150818708:task-definition/gost-staging-consume_grants:18" -> (known after apply)
      ~ arn_without_revision     = "arn:aws:ecs:us-west-2:357150818708:task-definition/gost-staging-consume_grants" -> (known after apply)
      ~ container_definitions    = (sensitive value) # forces replacement
      ~ id                       = "gost-staging-consume_grants" -> (known after apply)
-       ipc_mode                 = "" -> null
-       pid_mode                 = "" -> null
      ~ revision                 = 18 -> (known after apply)
-       tags                     = {} -> null
        # (9 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

  # module.postgres.module.db.aws_security_group_rule.this["from_arpa_audit_report"] must be replaced
-/+ resource "aws_security_group_rule" "this" {
      ~ id                       = "sgrule-1893596460" -> (known after apply)
      ~ security_group_rule_id   = "sgr-07df3250ab8676db1" -> (known after apply)
      ~ source_security_group_id = "sg-04836a204184dee25" -> (known after apply) # forces replacement
        # (6 unchanged attributes hidden)
    }

  # module.api.module.efs_data_volume.module.security_group.aws_security_group_rule.keyed["_m[0]#in#sg#1"] must be replaced
+/- resource "aws_security_group_rule" "keyed" {
      ~ id                       = "sgrule-203751513" -> (known after apply)
      ~ security_group_rule_id   = "sgr-0e1e7ae1a6ee1eaf7" -> (known after apply)
      ~ source_security_group_id = "sg-04836a204184dee25" -> (known after apply) # forces replacement
        # (8 unchanged attributes hidden)
    }

Plan: 9 to add, 3 to change, 9 to destroy.

Pusher: @TylerHendrickson, Action: pull_request, Workflow: Terraform CI