IAM errors when extending ECS Deploy Runner with AWS Systems Manager permissions

[zackproser]

A customer asked:

We attached the following policy to our allow-ops-admin-access-from-other-accounts role, since this is the role we assume when accessing other accounts.

... other permissions omitted for brevity ... 
 {
            "Sid": "SSM",
            "Effect": "Allow",
            "Action": [
                "ssm:AddTagsToResource",
                "ssm:GetParameter",
                "ssm:GetParameters",
                "ssm:PutParameter",
                "ssm:DeleteParameter",
                "ssm:RemoveTagsFromResource"
            ],
            "Resource": "*"
        },

The error we receive when attempting to launch sysdig into our account is:

│ Error: error creating SSM parameter (sysdig-api-token): AccessDeniedException: User: arn:aws:sts::[hidden]:assumed-role/allow-ops-admin-access-from-other-accounts/[hiddent] is not authorized to perform: ssm:PutParameter on resource: arn:aws:ssm:us-west-2:[hidden]:parameter/sysdig-api-token because no identity-based policy allows the ssm:PutParameter action
│     status code: 400, request id: [hidden]

[zackproser]

It looks like the allow-ops-admin-access-from-other-accounts role, by default, is configured with the same permissions that our Elastic Deploy Runner in the security account's mgmt folder shows in the deploy_permissions.yml file.

In most cases, this includes KMS:*, which would in turn include actions such as KMS:Decrypt and KMS:Encrypt. Why are the KMS permissions relevant?

In a couple of cases, even if you have sufficient permissions to take an action in IAM, but that action (say, ssm:PutParameter in this case) will end up making use of a KMS key, whether it be customer managed or a default one managed by AWS, then you still need sufficient permissions for that KMS key, too.

Otherwise, you get these really opaque and confusing access denied exceptions. Your policy above doesn't include the KMS:Decrypt or KMS:Encrypt actions, for example. I can see you do have the ssm:PutParameter action, hence I understand why you'd expect the action to be allowed. But if you have a look at the System Manager docs for the parameter store, you'll see you are going to require both of those permissions:

• SSM calls out to a KMS key to encrypt your secret
• SSM calls out to the same KMS key to decrypt your secret when you (or an AWS service) retrieve it

One other tip for debugging these kinds of things in the future: You can always check in your account's CloudTrail event history and see if there are any API calls around KMS, grants, etc that failed due to permissions issues.

So, tldr, please try adding KMS:Encrypt and KMS:Decrypt permissions to your inline policy and try your apply again.