kafka observe failed: failed to describe Cluster: AccessDeniedException

[csantanapr]

What happened?

Can't create simple kafka cluster

How can we reproduce it?

Using Composition from book platform engineering from @salaboy https://github.com/salaboy/platforms-on-k8s/blob/main/chapter-5/aws/resources/app-messagebroker-kafka.yaml

Results in MR like

apiVersion: kafka.aws.crossplane.io/v1alpha1
kind: Cluster
metadata:
  annotations:
    crossplane.io/composition-resource-name: kafka
    crossplane.io/external-name: aws-mb-kafka
  labels:
    crossplane.io/claim-name: ""
    crossplane.io/claim-namespace: ""
    crossplane.io/composite: aws-mb-kafka
  name: aws-mb-kafka-kafka
spec:
  deletionPolicy: Delete
  forProvider:
    brokerNodeGroupInfo:
      instanceType: kafka.t3.small
      storageInfo:
        ebsStorageInfo:
          volumeSize: 1
    clusterName: kafka
    kafkaVersion: 2.6.1
    numberOfBrokerNodes: 2
    region: us-west-2
  managementPolicies:
  - '*'
  providerConfigRef:
    name: default
  writeConnectionSecretToRef:
    name: kafka
    namespace: crossplane-system
status:
  atProvider: {}
  conditions:
  - lastTransitionTime: "2023-09-17T12:53:26Z"
    message: "observe failed: failed to describe Cluster: AccessDeniedException: \n\tstatus
      code: 403, request id: "
    reason: ReconcileError
    status: "False"
    type: Synced

Error:

2023-09-17T13:14:33.741Z        DEBUG   provider-aws    Cannot observe external resource    {"controller": "managed/cluster.kafka.aws.crossplane.io", "request": {"name":"aws-mb-kafka-kafka"}, "uid": "e20e52ba-0c80-41e0-b4bd-aa6d8e0a03ce", "version": "539545", "external-name": "aws-mb-kafka", "error": "failed to describe Cluster: AccessDeniedException: \n\tstatus code: 403, request id: ", "errorVerbose": "AccessDeniedException: \n\tstatus code: 403, request id: \ngithub.com/crossplane-contrib/provider-aws/pkg/clients.Wrap\n\tgithub.com/crossplane-contrib/provider-aws/pkg/clients/error.go:47

What environment did it happen in?

Crossplane version: 0.43.1

It's not a problem with IAM role being used, I'm creating redis and rds fine with a Role that has admin access

[csantanapr]

There is a problem setting the toFieldPath: metadata.annotations[crossplane.io/external-name]

[csantanapr]

got around the issue, but there might be a problem on observe only scenario

[MisterMX]

I am going to close this for now as this seems be an IAM misconfiguration that is not related to the controller itself. We have several different controllers reconciling MSK clusters without any issues.

Feel free to reopen if something changes.