Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to assume CDK lookup role using temporary session credentials #62

Open
tranhl opened this issue Jun 22, 2024 · 13 comments
Open

Unable to assume CDK lookup role using temporary session credentials #62

tranhl opened this issue Jun 22, 2024 · 13 comments
Labels
bug Something isn't working documentation Improvements or additions to documentation good first issue Good for newcomers help wanted Extra attention is needed

Comments

@tranhl
Copy link

tranhl commented Jun 22, 2024

Having a bit of trouble getting this action to work. Using the following configuration:

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v2
        with:
          aws-region: ${{ vars.AWS_REGION }}
          role-to-assume: ${{ secrets.ROLE_TO_ASSUME }}
          role-session-name: ci-session

      - name: Synth
        working-directory: apps/cdk
        run: pnpm cdk synth

      - name: Diff
        uses: corymhall/cdk-diff-action@v1
        with:
          failOnDestructiveChanges: false
          cdkOutDir: apps/cdk/cdk.out
          githubToken: ${{ secrets.GITHUB_TOKEN }}

I get the following error:

AccessDenied: User: arn:aws:sts::***:assumed-role/github-action/ci-session is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::${AWS::AccountId}:role/cdk-hnb659fds-lookup-role-${AWS::AccountId}-${AWS::Region}

Seems like the issue is that AWS::AccountId and AWS::Region isn't templating correctly when assuming the CDK lookup role? Not exactly sure why that would be the case. I've included the full error logs, happy to provide additional information needed.

Full error log
Error performing stack diff:  AccessDenied: User: arn:aws:sts::***:assumed-role/github-action/monorepoCISession is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::${AWS::AccountId}:role/cdk-hnb659fds-lookup-role-${AWS::AccountId}-${AWS::Region}
    at throwDefaultError (/home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@smithy/smithy-client/dist-cjs/index.js:838:1)
Error: User: arn:aws:sts::***:assumed-role/github-action/monorepoCISession is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::${AWS::AccountId}:role/cdk-hnb659fds-lookup-role-${AWS::AccountId}-${AWS::Region}
    at /home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@smithy/smithy-client/dist-cjs/index.js:847:1
    at de_CommandError (/home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@aws-sdk/client-sts/dist-cjs/index.js:478:1)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at /home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:1
    at /home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@smithy/core/dist-cjs/index.js:165:1
    at /home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@smithy/middleware-retry/dist-cjs/index.js:3[20](https://github.com/<redacted>/monorepo/actions/runs/9616955847/job/26527645521?pr=854#step:6:21):1
    at /home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:34:1
    at /home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@aws-sdk/credential-providers/dist-cjs/fromTemporaryCredentials.js:47:1
    at coalesceProvider (/home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@smithy/core/dist-cjs/index.js:288:1) {
  '$fault': 'client',
  '$metadata': {
    httpStatusCode: 403,
    requestId: '3092bae5-14f9-49e6-84bc-ce1ae49[22](https://github.com/<redacted>/monorepo/actions/runs/9616955847/job/26527645521?pr=854#step:6:23)3d8',
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  Type: 'Sender',
  Code: 'AccessDenied'
}
Error processing stages:  AccessDenied: User: arn:aws:sts::***:assumed-role/github-action/monorepoCISession is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::${AWS::AccountId}:role/cdk-hnb659fds-lookup-role-${AWS::AccountId}-${AWS::Region}
    at throwDefaultError (/home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@smithy/smithy-client/dist-cjs/index.js:838:1)
    at /home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@smithy/smithy-client/dist-cjs/index.js:847:1
    at de_CommandError (/home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@aws-sdk/client-sts/dist-cjs/index.js:478:1)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at /home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:1
    at /home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@smithy/core/dist-cjs/index.js:165:1
    at /home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:1
    at /home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:34:1
    at /home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@aws-sdk/credential-providers/dist-cjs/fromTemporaryCredentials.js:47:1
    at coalesceProvider (/home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@smithy/core/dist-cjs/index.js:288:1) {
  '$fault': 'client',
  '$metadata': {
    httpStatusCode: 403,
    requestId: '3092bae5-14f9-49e6-84bc-ce1ae492[23](https://github.com/<redacted>/monorepo/actions/runs/9616955847/job/26527645521?pr=854#step:6:24)d8',
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  Type: 'Sender',
  Code: 'AccessDenied'
}
Error running process stages:  AccessDenied: User: arn:aws:sts::***:assumed-role/github-action/monorepoCISession is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::${AWS::AccountId}:role/cdk-hnb659fds-lookup-role-${AWS::AccountId}-${AWS::Region}
    at throwDefaultError (/home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@smithy/smithy-client/dist-cjs/index.js:838:1)
    at /home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@smithy/smithy-client/dist-cjs/index.js:847:1
    at de_CommandError (/home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@aws-sdk/client-sts/dist-cjs/index.js:478:1)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at /home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:1
    at /home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@smithy/core/dist-cjs/index.js:165:1
    at /home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:1
    at /home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:34:1
    at /home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@aws-sdk/credential-providers/dist-cjs/fromTemporaryCredentials.js:47:1
    at coalesceProvider (/home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@smithy/core/dist-cjs/index.js:[28](https://github.com/<redacted>/monorepo/actions/runs/9616955847/job/26527645521?pr=854#step:6:29)8:1) {
  '$fault': 'client',
  '$metadata': {
    httpStatusCode: 403,
    requestId: '3092bae5-14f9-49e6-84bc-ce1ae49223d8',
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  Type: 'Sender',
  Code: 'AccessDenied'
}
Error performing diff:  AccessDenied: User: arn:aws:sts::***:assumed-role/github-action/monorepoCISession is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::${AWS::AccountId}:role/cdk-hnb659fds-lookup-role-${AWS::AccountId}-${AWS::Region}
    at throwDefaultError (/home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@smithy/smithy-client/dist-cjs/index.js:838:1)
    at /home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@smithy/smithy-client/dist-cjs/index.js:847:1
    at de_CommandError (/home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@aws-sdk/client-sts/dist-cjs/index.js:478:1)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at /home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:1
    at /home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@smithy/core/dist-cjs/index.js:165:1
    at /home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:1
    at /home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:34:1
    at /home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@aws-sdk/credential-providers/dist-cjs/fromTemporaryCredentials.js:47:1
    at coalesceProvider (/home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@smithy/core/dist-cjs/index.js:288:1) {
  '$fault': 'client',
  '$metadata': {
    httpStatusCode: 403,
    requestId: '[30](https://github.com/<redacted>/monorepo/actions/runs/9616955847/job/26527645521?pr=854#step:6:31)92bae5-14f9-49e6-84bc-ce1ae49223d8',
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  Type: 'Sender',
  Code: 'AccessDenied'
}
@tranhl tranhl changed the title AccessDenied: User: arn:aws:sts::***:assumed-role/github-action/ci-session is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::${AWS::AccountId}:role/cdk-hnb659fds-lookup-role-${AWS::AccountId}-${AWS::Region} Unable to assume CDK lookup role using temporary session credentials Jun 22, 2024
@tranhl
Copy link
Author

tranhl commented Jun 24, 2024

Turns out the issue is that AWS::AccountId and AWS::Region won't have an actual value in the CDK assembly unless we explicitly pass a value to props.env on all our stacks during synthesis. The action worked after env was provided.

@corymhall
Copy link
Owner

@tranhl sorry you ran into this error, we should probably update the readme to note that you have to specify the env for this action to work, otherwise there is no way for it to know which account/region is the target.

@corymhall corymhall added the documentation Improvements or additions to documentation label Sep 5, 2024
@rantoniuk
Copy link

rantoniuk commented Nov 6, 2024

@corymhall I think I stumbled on this as well but I'm not sure that I agree with the solution. All my stacks are working fine on the local console using AWS_PROFILE= and AWS SSO temporary credentials, the cdk synth command generates a generic manifest, that is environment-agnostic:

cat cdk.out/manifest.json| grep acc     
      "environment": "aws://unknown-account/unknown-region",
      "environment": "aws://unknown-account/unknown-region",
      "environment": "aws://unknown-account/unknown-region",

This is in-line with the actual documentation of the behaviour from CDK - example here. I do not need/want to make my stacks non-account agnostic.

Github Action

Then in my GH action setup, I do have an IAM Role that is successfully assumed via OIDC:

Run aws-actions/configure-aws-credentials@v4
Assuming role with OIDC
Authenticated as assumedRoleId AROAZ4SRUYMZEW:github-infra-cdk-diff

and I have granted this role the CloudFormationReadOnlyAccess AWS Managed policy that should be enough to perform the diff.

However the action is trying to assume the lookup role instead of the temporary credentials obtained via OIDC:

Error performing stack diff:  AccessDenied: User: arn:aws:sts::xxx:assumed-role/githubActionsDeployRole/github-infra-cdk-diff is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::${AWS::AccountId}:role/cdk-hnb659fds-lookup-role-${AWS::AccountId}-${AWS::Region}

I would say the logic here should be exactly the opposite, use the temporary credentials if they are present (or maybe that could be a flag?)

EDIT: Before running this action, I tried to run cdk diff in the workflow and even though it runs without any error, it also shows the same problem of

current credentials could not be used to assume 'arn:aws:iam::xxx:role/cdk-hnb659fds-lookup-role-xxx-us-west-2', but are for the right account. Proceeding anyway.

I'll keep investigating why this does work on a local CLI with agnostic stacks but doesn't work with temporary credentials on the handcrafted IAM role.

@corymhall
Copy link
Owner

@rantoniuk this guide here has some additional details about how the CDK gets credentials for things. The CDK CLI (and this action) will only use your credentials to assume-role into the role that actually has the permissions to perform the action.

https://github.com/aws/aws-cdk/wiki/Security-And-Safety-Dev-Guide#defaultstacksynthesizer

@rantoniuk
Copy link

I have fixed the problem of the manual cdk diff in the pipeline, by adding the required IAM sts:AssumeRole permissions:

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Effect": "Allow",
			"Action": "sts:AssumeRole",
			"Resource": "arn:aws:iam::*:role/cdk-hnb659fds-*"
		}
	]
}

to the role that is assumed in the OIDC step:

      - name: Authenticate Via OIDC Role
        uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-region: ${{ vars.AWS_REGION }}
          role-duration-seconds: 900
          role-to-assume: ${{ vars.AWS_OIDC_IAM_DEPLOY_ROLE }}
          role-session-name: github-infra-cdk-diff

and now I see exactly the same output as when I'm doing a diff locally on the CLI console.

Yet, in the next step, this action still has the same issue as before, trying to assume a role without expanding the AccountID placeholder:

Error performing stack diff:  AccessDenied: User: arn:aws:sts::679849022850:assumed-role/githubActionsDeployRole/github-infra-cdk-diff is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::${AWS::AccountId}:role/cdk-hnb659fds-lookup-role-${AWS::AccountId}-${AWS::Region}

I'm going to try to build a custom version of the action with the swapped logic to see if that fixes the issue.

@rantoniuk
Copy link

@corymhall the action works flawlessly when I removed the custom credentials logic and disabled associated tests and tested via
uses: rantoniuk/corymhall-cdk-diff-action@use-default-credentials

In the commented PR I see wrong warning:

This stack has an unknown environment which may mean the diff is performed against the wrong environment

that is also coming from the action logic. Take a look at the changes and let me know if you think we could incorporate them into your action. I think that should be the default behaviour but I'd be also happy with a flag.

Happy to provide a PR!

@corymhall
Copy link
Owner

@rantoniuk thanks for digging into this, I think I'm finally seeing what the issue is. There are two cases that are hard to tell apart and we currently only support one of them

  1. User has stacks and each stack deploys to a different environment
  2. User has stacks and they all deploy to the same environment (not handling this one)

In the first case the user has to provide the environment otherwise there is no way to tell which environment to perform the diff for.

We just need to handle the second case. When we get the credentials here:

credentials = stack.lookupRole ? fromTemporaryCredentials({
params: {
RoleArn: stack.lookupRole.arn.replace('${AWS::Partition}', this.getPartition()),
RoleSessionName: 'cdk-diff-action',
ExternalId: stack.lookupRole.assumeRoleExternalId,
DurationSeconds: 900,
},
}) : undefined;
this.credentials = credentials;
this.client = new CloudFormationClient({
credentials,
region: this.stack.region,
});

If the user provides the lookup role and the arn has an unknown account/region we should just substitute the current account/region (from the current credentials) like we do for the partition. We should also still log a warning to let the user know that we are making this assumption because there is a chance that they should have provided the environment, but have a misconfiguration.

I won't be able to pick this one up in the short term, but would definitely take a PR!

@corymhall corymhall added bug Something isn't working help wanted Extra attention is needed good first issue Good for newcomers labels Nov 11, 2024
@rantoniuk
Copy link

rantoniuk commented Nov 11, 2024

@corymhall I agree about the unsupported case, but I fail to understand why would this action need to support anything at all instead of relying on the credentials provided in earlier pipeline steps.

The purpose of this action could/should be just to interpret the diff based on cdk synth's output that is done in earlier pipeline steps. That being said, it should not need to mingle with any credentials and even if the action itself would like to run the diff itself via the DefaultSynthesizer, then it should still rely on properly configured credentials before.

Whether the stack is environment-agnostic or not, is not to be decided by this action, but instead by the Stack configuration.

The reason my version works is because:

  1. my stacks are environment agnostic, meaning that the IaaC is universal and I'm deploying it to many environments
  2. I have already assumed in the pipeline a dedicated role:
      - name: Authenticate Via OIDC Role
        uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-region: ${{ vars.AWS_REGION }}
          role-duration-seconds: 900
          role-to-assume: ${{ vars.AWS_OIDC_IAM_DEPLOY_ROLE }}
          role-session-name: github-infra-cdk-diff
  1. and that the ${{ vars.AWS_OIDC_IAM_DEPLOY_ROLE }} role has the necessary permission:
{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Effect": "Allow",
			"Action": "sts:AssumeRole",
			"Resource": "arn:aws:iam::*:role/cdk-hnb659fds-*"
		}
	]
}

to assume the corresponding lookup roles in the target account that are defined in the manifest.json output. (Here the target account is indeed the same account but that does not matter)

In other words, responding to your comment - I'm not providing any lookup roles explicitly, everything is handled by CDK itself.

For the 1) supported case User has stacks and each stack deploys to a different environment. In such case, the account configuration comes from either env configuration or from making the stack account-specific by hardcoding the account numbers into the stack.

My underlying logic here is: as long as cdk diff works as a manual step somewhere earlier in the pipeline, this action should work as well without any additional logic.

@corymhall
Copy link
Owner

The purpose of this action could/should be just to interpret the diff based on cdk synth's output that is done in earlier pipeline steps. That being said, it should not need to mingle with any credentials and even if the action itself would like to run the diff itself via the DefaultSynthesizer, then it should still rely on properly configured credentials before.

Whether the stack is environment-agnostic or not, is not to be decided by this action, but instead by the Stack configuration.

I think we are saying similar things. This action is trying to mimic the behavior of the CDK CLI. When you run cdk diff from the CLI, the CLI will read the CloudAssembly (cdk.out) and if you are using the DefaultStackSynthesizer it will assume the necessary role in the target account to perform the diff. When you run cdk diff it will not just use whatever credentials you have configured locally, it will always assume the specified roles in the CloudAssembly.

This action attempts to do the same thing. When it performs the diff it reads the CloudAssembly and if you are using the DefaultStackSynthesizer then it will assume the specified role to perform the diff.

The reason you are running into this error in this action, but not when using the CLI is because the CLI just has more robust credential logic and this action suffers from the bug I mentioned in my last comment.

@rantoniuk
Copy link

This action is trying to mimic the behavior of the CDK CLI

Is there any specific reason it does that instead of relying on CDK buil-in capabilities? After all, you anyway are running npx cdk synth before via CLI to create the CloudAssembly.

The reason you are running into this error in this action, but not when using the CLI is because the CLI just has more robust credential logic and this action suffers from the bug I mentioned in my last comment.

Again, from my perspective the bug is that it implements the custom logic in the first place, because when this is removed, everything works fine with the role that was assumed in the previous step and to which the short-lived credentials are present.

@corymhall
Copy link
Owner

@rantoniuk the custom logic exists because I want more control over the output. Because of the custom logic this action is able to do things like highlight resources that have destructive changes, fail the workflow if there are destructive changes, filter on what resources to allow destructive changes on, etc.

If all you want is the raw diff output from cdk diff this action is probably not needed and something simpler would suffice.

@rantoniuk
Copy link

rantoniuk commented Nov 14, 2024

Unfortunately this does not answer my question at all. What you're describing is post-authentication logic and this whole discussion is about why the custom logic exists for authentication:

let credentials: AwsCredentialIdentityProvider | undefined;
// if there is a lookup role then assume that, otherwise
// just use the default credentials
credentials = stack.lookupRole ? fromTemporaryCredentials({
params: {
RoleArn: stack.lookupRole.arn.replace('${AWS::Partition}', this.getPartition()),
RoleSessionName: 'cdk-diff-action',
ExternalId: stack.lookupRole.assumeRoleExternalId,
DurationSeconds: 900,
},
}) : undefined;
this.credentials = credentials;

I think I explained my view above already and I don't want to pollute the issue. From my POV if you are fine with adding a flag to use the default credentials, then I'm happy to do it, but I don't see any point of implementing yet more complex custom logic to cover the use case in the action itself, because it shouldn't be here imo.

@corymhall
Copy link
Owner

@rantoniuk

When you perform cdk diff this is roughly what is happening.

Screenshot 2024-11-14 at 1 06 19 PM

There is no ability to pick and choose which things you want and which you do not. In my case for this action, I want to change the very last thing. I want to take the output of fullDiff and do something else with it. In order to do that I have to replace the entire box with custom logic, and this includes the authentication piece.

This is probably more information than you care for, but it's why I don't want to add the flag that you are suggesting. Anyone should definitely feel free to fork this action and customize it to work better for you individual use case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working documentation Improvements or additions to documentation good first issue Good for newcomers help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

3 participants