AccessKey credentials are reset on every cluster creation

[jbpratt]

Access keys are being reset on every rosa create cluster ... call such that subsequent calls (rosa list clusters) failed due to invalid credentials. These credentials are also used when working with the ocm CLI so it is causing issues for my team since we store these. Would it be possible to not upsert the access keys on cluster creation or perhaps suggest an alternative workflow? cc'ing @jharrington22 since they wrote the code and explanation.

rosa/pkg/aws/client.go

Lines 471 to 494 in 7b3efee

	// GetAWSAccessKeys uses UpsertAccessKey to delete and create new access keys
	// for `osdCcsAdmin` each time we use the client to create a cluster.
	// There is no need to permanently store these credentials since they are only used
	// on create, the cluster uses a completely different set of IAM credentials
	// provisioned by this user.
	func (c *awsClient) GetAWSAccessKeys() (*AccessKey, error) {
	if c.awsAccessKeys != nil {
	return c.awsAccessKeys, nil
	}
	accessKey, err := c.UpsertAccessKey(AdminUserName)
	if err != nil {
	return nil, err
	}
	err = c.ValidateAccessKeys(accessKey)
	if err != nil {
	return nil, err
	}
	c.awsAccessKeys = accessKey
	return c.awsAccessKeys, nil
	}

❯ rosa list clusters
E: Failed to create AWS client: InvalidClientTokenId: The security token included in the request is invalid.
	status code: 403, request id: 77591dc5-8d54-4282-9ec9-b20xyzabc658eb

[yuwang-RH]

@jbpratt I think this issue will not happen if you configure aws credential with another aws user rather than the 'osdCCSadmin' user credential.

[yuwang-RH]

@jbpratt I have opened a JIRA issue about this, https://issues.redhat.com/browse/SDA-5644

[jbpratt]

thanks for that @yuwang-RH, I think an issue with using another user is that OCM utilizes this user, so while subsequent calls to ROSA stop working, OCM does as well. According to the docs, this is the user to use. (Note: we are trying to use OCM and ROSA in the same account)

[arendej]

I don't mean to side-track the issue, but, why not use STS mode?

[jbpratt]

Hey @arendej , thanks for the recommendation, I will need to look more into it. I think even if we use an alternate auth method, the credentials are still rotated which causes issues with the ocm cli.

[jbpratt]

Hey @arendej what are your thoughts on the rosa CLI upsert conflicting with the ocm CLI? Maybe I'm doing something wrong but it seems I can't easily work with both tools in the same account.

edit: maybe if I'm using STS mode then the credentials don't get upserted? I'm looking into this now

[jbpratt]

that is exactly what it is! 😺 thank you very much @arendej and @yuwang-RH ! STS mode doesn't upsert the creds 👍

[jbpratt]

Going to re-open this issue, various code paths (rosa init?) seem to reset credentials which is causing issues with our OSD deployments.

[trepel]

Hey, I encountered this issue too. STS is not an option for us since our product does not support STS just yet. If you create a ROSA cluster in certain aws account (not matter what aws user is used) the ROSA cli upsert the osdCcsAdmin's credentials. In order to use the aws account for "standard" OSD creation one has to create a new keypair for osdCcsAdmin first which is not really feasible for automation.

[mrrajan]

Hi, I have faced encountered this as well. For each ROSA cluster creation, the existing osdCcsAdmin aws access keys pairs are removed and this impacts the usage of access key pairs in other parts of automation to provision OSD clusters.

[jbpratt]

Hey @arendej, are there any plans to resolve this problem? It seems this issue is leaking into other features as well cc @oriAdler 1299a60

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[jbpratt]

/lifecycle frozen