ScaledObject Fails to resolve Auth params from cluster scoped ClusterTriggerAuth resource

[sreeboppana]

Report

Keda version: keda:2.14.0
ghcr.io/kedacore/keda:2.14.0
kubenetes version: 1.29
Keda Trigger Type: Kafka

ScaledObject Fails [namespace scoped] to resolve Auth params from cluster scoped ClusterTriggerAuth resource

Expected Behavior

Keda ScaledObject ideally should have read the credentials from the ClusterTriggerAuth resource and authenticate with the KAfka API.

Actual Behavior

2024-11-06T16:49:43Z  ERROR scale_handler error resolving auth params {"type": "ScaledObject", "namespace": "my-app-namespace", "name": "my-app-microservice", "triggerIndex": 4, "error": "error creating kafka client: kafka: client has run out of available brokers to talk to: read tcp 172.22.199.48:59274->100.66.23.62:9092: read: connection reset by peer"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).buildScalers
  /workspace/pkg/scaling/scalers_builder.go:99
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).performGetScalersCache
  /workspace/pkg/scaling/scale_handler.go:357
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScalersCache
  /workspace/pkg/scaling/scale_handler.go:282
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).getScaledObjectState
  /workspace/pkg/scaling/scale_handler.go:611
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
  /workspace/pkg/scaling/scale_handler.go:243
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
  /workspace/pkg/scaling/scale_handler.go:182

2024-11-06T16:49:43Z  ERROR scale_handler error getting state of scaledObject {"scaledObject.Namespace": "my-app-namespace", "scaledObject.Name": "my-app-microservice", "error": "error getting scalers cache error creating kafka client: kafka: client has run out of available brokers to talk to: read tcp 172.22.199.48:59274->100.66.23.62:9092: read: connection reset by peer"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
  /workspace/pkg/scaling/scale_handler.go:245
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
  /workspace/pkg/scaling/scale_handler.go:182

Steps to Reproduce the Problem

STEP-1 Ensured that k8s Secret - keda-kafka-secret created within the keda namespace.

% kubectl get secret keda-kafka-secret -oyaml -nkeda

apiVersion: v1
data:
  keda-kafka-api-key: KAFKA-API-KEY
  keda-kafka-api-secret: KAFKA-API-SECRET
kind: Secret
type: Opaque
metadata:
  creationTimestamp: "2024-10-31T08:06:59Z"
  labels:
    secrets-store.csi.k8s.io/managed: "true"
  managedFields:
  - apiVersion: v1
  name: keda-kafka-secret
  namespace: keda

STEP-2. Created a cluster scoped ClusterTriggerAuthentication resource & refer to the
secret from STEP-2 which is keda namespace scoped.

apiVersion: keda.sh/v1alpha1
kind: ClusterTriggerAuthentication
metadata:
  name: keda-kafka-clustertriggerauth
spec:
  secretTargetRef:
  - parameter: keda-kafka-api-key
    name: keda-kafka-secret
    key: keda-kafka-api-key
  - parameter: keda-kafka-api-secret
    name: keda-kafka-secret
    key: keda-kafka-api-secret

STEP-3. Create a ScaledObject within my-app-namespace namespace referring to the above cluster scoped ClusterTriggerAuthentication to read the credentials & authenticate with Kafka (kafka trigger).

---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: kafka-scaledobject
  namespace: my-app-namespace
spec:
  scaleTargetRef:
    name: my-demo-app-deployment
  pollingInterval: 30
  triggers:
  - type: kafka
    metadata:
      bootstrapServers: kafkaHost:9092
      consumerGroup: my-consumer-group
      topic: kafka-demo-topic
      lagThreshold: "30"
    authenticationRef:
      name: keda-kafka-clustertriggerauth
      kind: ClusterTriggerAuthentication

STEP-4 Just describe keda-operator pod (informational purpose).

% kubectl describe po keda-operator-5b48dfb687-b7xcx -n keda

Name:         keda-operator-5b48dfb687-b7xcx
Namespace:    keda
Priority:     0
Node:         ip-100-66-9-70.ec2.internal/100.66.9.70
Start Time:   Tue, 05 Nov 2024 07:40:10 -0500
Labels:       app=keda-operator
              app.kubernetes.io/component=operator
              app.kubernetes.io/instance=keda-operator
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=keda-operator
              app.kubernetes.io/part-of=keda-operator
              app.kubernetes.io/version=2.14.0
              helm.sh/chart=keda-2.14.2
              name=keda-operator
              pod-template-hash=5b48dfb687
Annotations:  kubectl.kubernetes.io/restartedAt: 2024-10-31T09:06:59+01:00
Status:       Running
IP:           172.22.199.48
IPs:
  IP:           172.22.199.48
Controlled By:  ReplicaSet/keda-operator-5b48dfb687
Containers:
  keda-operator:
    Container ID:  containerd://94334ff68a5dab55f2c32a1f10a54230eaa2139ff2654fa20c858cf5b31c5dc8
    Image:         ghcr.io/kedacore/keda:2.14.0
    Image ID:      ghcr.io/kedacore/keda@sha256:c74847dd7c2a62d6a3ad0631208dab53264cec4eb0fc5ac3f41d4852ee363bf6
    Ports:         8080/TCP, 9666/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      /keda
    Args:
      --leader-elect
      --disable-compression=true
      --zap-log-level=info
      --zap-encoder=console
      --zap-time-encoding=rfc3339
      --cert-dir=/certs
      --enable-cert-rotation=true
      --cert-secret-name=kedaorg-certs
      --operator-service-name=keda-operator
      --metrics-server-service-name=keda-operator-metrics-apiserver
      --webhooks-service-name=keda-admission-webhooks
      --k8s-cluster-name=kubernetes-default
      --k8s-cluster-domain=cluster.local
      --enable-prometheus-metrics=true
      --metrics-bind-address=:8080
    State:          Running
      Started:      Tue, 05 Nov 2024 07:40:13 -0500
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  8Gi
    Requests:
      cpu:      1
      memory:   8Gi
    Liveness:   http-get http://:8081/healthz delay=25s timeout=1s period=10s #success=1 #failure=3
    Readiness:  http-get http://:8081/readyz delay=20s timeout=1s period=3s #success=1 #failure=3
    Environment:
      WATCH_NAMESPACE:              
      POD_NAME:                     keda-operator-5b48dfb687-b7xcx (v1:metadata.name)
      POD_NAMESPACE:                keda (v1:metadata.namespace)
      OPERATOR_NAME:                keda-operator
      KEDA_HTTP_DEFAULT_TIMEOUT:    3000
      KEDA_HTTP_MIN_TLS_VERSION:    TLS12
      AWS_STS_REGIONAL_ENDPOINTS:   regional
      AWS_DEFAULT_REGION:           us-east-1
      AWS_REGION:                   us-east-1
      AWS_ROLE_ARN:                 arn:aws:iam::12345679:role/keda-operator-iam-role-us-east-1
      AWS_WEB_IDENTITY_TOKEN_FILE:  /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    Mounts:
      /certs from certificates (ro)
      /mnt/kafka-secret from kafka-secret (ro)
      /var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-m9vcm (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       True 
  ContainersReady             True 
  PodScheduled                True 
Volumes:
  aws-iam-token:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  86400
  certificates:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  kedaorg-certs
    Optional:    true
  kafka-secret:
    Type:              CSI (a Container Storage Interface (CSI) volume source)
    Driver:            secrets-store.csi.k8s.io
    FSType:            
    ReadOnly:          true
    VolumeAttributes:      secretProviderClass=keda-kafka-secretproviderclass
  kube-api-access-m9vcm:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>

Logs from KEDA operator

2024-11-06T16:49:43Z  ERROR scale_handler error resolving auth params {"type": "ScaledObject", "namespace": "my-app-namespace", "name": "my-app-microservice", "triggerIndex": 4, "error": "error creating kafka client: kafka: client has run out of available brokers to talk to: read tcp 172.22.199.48:59274->100.66.23.62:9092: read: connection reset by peer"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).buildScalers
  /workspace/pkg/scaling/scalers_builder.go:99
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).performGetScalersCache
  /workspace/pkg/scaling/scale_handler.go:357
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScalersCache
  /workspace/pkg/scaling/scale_handler.go:282
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).getScaledObjectState
  /workspace/pkg/scaling/scale_handler.go:611
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
  /workspace/pkg/scaling/scale_handler.go:243
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
  /workspace/pkg/scaling/scale_handler.go:182

2024-11-06T16:49:43Z  ERROR scale_handler error getting state of scaledObject {"scaledObject.Namespace": "my-app-namespace", "scaledObject.Name": "my-app-microservice", "error": "error getting scalers cache error creating kafka client: kafka: client has run out of available brokers to talk to: read tcp 172.22.199.48:59274->100.66.23.62:9092: read: connection reset by peer"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
  /workspace/pkg/scaling/scale_handler.go:245
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
  /workspace/pkg/scaling/scale_handler.go:182

KEDA Version

2.14.0

Kubernetes Version

1.29

Platform

Amazon Web Services

Scaler Details

Kafka

Anything else?

No response

[sreeboppana]

@JorTurFer can you please advise on this please ?

[apoole-q6cyber]

This might be a question for stackoverflow

[JorTurFer]

Sorry for the delay.
Is your secret in KEDA's namespace? When you use a ClusterTriggerAuthentication, the secrets can be only in the namespace where KEDA is.