[Bug]: Kraft migration with AWS external loadbalancer gets stuck

[jeromeguiard]

Bug Description

While triggering a migration from zookeeper to Kraft. The controller nodepool, newly created, never gets healthy. The strimzi operator keeps reporting timeout on newly created nodes.

This happens with a bootstrap config that registers the node to an AWS load balancer.

ome ready: io.strimzi.operator.common.TimeoutException: Exceeded timeout of 300000ms while waiting for Pods resource xxx-dev-poc-xxx-dev-poc-controller-4 in namespace kafka to be ready
2024-10-14 14:07:26 WARN  KafkaAgentClient:141 - Reconciliation #635(watch) Kafka(middleware-kafka/xxx-dev-poc): Failed to get broker state
java.lang.RuntimeException: Unexpected HTTP status code: 404
	at io.strimzi.operator.cluster.operator.resource.KafkaAgentClient.doGet(KafkaAgentClient.java:114) ~[io.strimzi.cluster-operator-0.43.0.jar:0.43.0]
	at io.strimzi.operator.cluster.operator.resource.KafkaAgentClient.getBrokerState(KafkaAgentClient.java:135) ~[io.strimzi.cluster-operator-0.43.0.jar:0.43.0]
	at io.strimzi.operator.cluster.operator.resource.KafkaRoller.restartIfNecessary(KafkaRoller.java:438) ~[io.strimzi.cluster-operator-0.43.0.jar:0.43.0]
	at io.strimzi.operator.cluster.operator.resource.KafkaRoller.lambda$schedule$7(KafkaRoller.java:368) ~[io.strimzi.cluster-operator-0.43.0.jar:0.43.0]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
2024-10-14 14:07:26 WARN  KafkaRoller:444 - Reconciliation #635(watch) Kafka(middleware-kafka/xxx-dev-poc): Pod xxx-dev-poc-xxx-dev-poc-controller-4 is not ready. We will check if KafkaRoller can do anything about it.

It appears that the controller node remains unhealthy because it can't become ready. I think it is due to it's registration to the load balancer that keep failing. Most likely the port 9092 is not listening, and so the readiness gate on the pod with the controller role never becomes ready from the operator side.

One fix i found was, in the kubernetes service handling the bootstrap description, to add in spec selector:

strimzi.io/broker-role: "true"

to ensure that only nodes with broker role will be included in the load balancer.

Once added, and pod restarted, the migration actually starts

Steps to reproduce

1. have a running kafka cluster with zookeeper, kraft disabled an active nodepool for broker, a nodepool for controller ready and this configuration for listener

    listeners:
    - configuration:
        bootstrap:
          annotations:
            service.beta.kubernetes.io/aws-load-balancer-scheme: internal
            service.beta.kubernetes.io/aws-load-balancer-subnets: subnet-XXX
            service.beta.kubernetes.io/aws-load-balancer-type: nlb-ip
        brokers:
        - annotations:
            service.beta.kubernetes.io/aws-load-balancer-scheme: internal
            service.beta.kubernetes.io/aws-load-balancer-subnets: subnet-XXX
            service.beta.kubernetes.io/aws-load-balancer-type: nlb-ip
          broker: 0
        - annotations:
            service.beta.kubernetes.io/aws-load-balancer-scheme: internal
            service.beta.kubernetes.io/aws-load-balancer-subnets: subnet-XXX
            service.beta.kubernetes.io/aws-load-balancer-type: nlb-ip
          broker: 1
        - annotations:
            service.beta.kubernetes.io/aws-load-balancer-scheme: internal
            service.beta.kubernetes.io/aws-load-balancer-subnets: subnet-XXX
            service.beta.kubernetes.io/aws-load-balancer-type: nlb-ip
          broker: 2
      name: plain
      port: 9092
      tls: false
      type: loadbalancer

2 . start a kraft migration

annotations:
  strimzi.io/kraft: migration

3. observer new nodepool being created and never become ready

Expected behavior

Expecting the migration to trigger as described in the doc

Strimzi version

0.43.0

Kubernetes version

EKS 1.30

Installation method

helm chart

Infrastructure

EKS

Configuration files and logs

No response

Additional context

No response

[scholzj]

The controller nodes do not do any load balancer registration. In fact, even brokers don't do anything like that. The error itself does not seem to make much sense as the pod is either not ready or we try to connect to it. But you should probably start by sharing full logs and configurations of all compoenents.

[scholzj]

PS: If you are using something like this https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.1/deploy/pod_readiness_gate/ to bind the Pod readiness to the loadbalancers, than you should make sure it is not applied to the controller nodes as they do not access client connections through any loadbalancers.

[jeromeguiard]

You are right to point it out.

thanks

[ppatierno]

Triaged on 17/10/2024: decided to convert into a discussion because it doesn't seem to be a Strimzi bug.