Scaling up brokers __consumer_offset replication factor doesn't increase version 0.44

[haroon9]

In Kraft mode, I scaled up the number of brokers from 1 to 3 and adjusted the config now all the topics have replication factor 3 but the __consumer_offset topic has replication factor 1. The cluster was rebalanced using the cruise control config. But using the UI I noticed that Broker ID 0 shows that the load distribution across cluster is not balanced.

Another thing I noticed is that after scaling up the cluster our client app still connects to the Group Coordinator on the old broker with ID 0 and not the new brokers and will complain if the broker ID 0 is not running. Is this due to the imbalance on the first broker and the __consumer_offset topic replication factor?

Kafka config:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: kafka
  labels:
    strimzi.io/cluster: kafka
  annotations:
    strimzi.io/kraft: enabled
    strimzi.io/node-pools: enabled
spec:
  kafka:
    version: 3.7.0
    logging:
      type: inline
      loggers:
        kafka.root.logger.level: INFO
    readinessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    livenessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    listeners:
      - name: tls
        port: 9092
        type: internal
        tls: true
        authentication:
          type: tls
      - name: ingress
        port: 9093
        type: ingress
        tls: true
        authentication:
          type: tls
        configuration:
          bootstrap:
            host: kafka-server-staging.carbonfreed.com
          brokers:
          - broker: 0
            host: broker-0-staging.carbonfreed.com
          - broker: 1
            host: broker-1-staging.carbonfreed.com
          - broker: 2
            host: broker-2-staging.carbonfreed.com
          class: nginx
    authorization:
        type: simple
    config:
      auto.leader.rebalance.enable: true
      auto.create.topics.enable: "false"
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      group.initial.rebalance.delay.ms: 3000
  entityOperator:
    topicOperator:
      watchedNamespace: kafka
      reconciliationIntervalMs: 60
      logging:
        loggers:
          rootLogger.level: INFO
        type: inline
    userOperator:
      watchedNamespace: kafka
      reconciliationIntervalMs: 60
      logging:
        loggers:
          rootLogger.level: INFO
        type: inline
  cruiseControl:
    autoRebalance:
      - mode: add-brokers
        template:
          name: add-brokers-rebalance
      - mode: remove-brokers
        template:
          name: remove-brokers-rebalance
---
# Source: kafka/templates/broker.yaml
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: kafka
  labels:
    strimzi.io/cluster: kafka
spec:
  replicas: 3
  roles:
    - broker
  storage:
    type: jbod
    volumes:
      - id: 0
        type: persistent-claim
        size: 10Gi
        kraftMetadata: shared
        deleteClaim: false
---
# Source: kafka/templates/controller.yaml
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: controller
  labels:
    strimzi.io/cluster: kafka
spec:
  replicas: 1
  roles:
    - controller
  storage:
    type: jbod
    volumes:
      - id: 0
        type: persistent-claim
        size: 5Gi
        kraftMetadata: shared
        deleteClaim: false
---
# Source: kafka/templates/topics.yaml
apiVersion: kafka.strimzi.io/v1beta1
kind: KafkaTopic
metadata:
  name: ai-analysis-requests.v0
  labels:
    strimzi.io/cluster: kafka
spec:
  partitions: 3
  replicas: 3
  config:
    retention.ms: 86400000
    segment.bytes: 52428800

Ideally, the backend client should connect to any other available broker in case the broker it was connected to before is down. Also, the client discovers all the brokers but log these when the first broker is not available:

{"timestamp": "2024-11-19T18:27:17.468257Z", "level": "ERROR", "message": "Group Coordinator Request failed: [Error 15] GroupCoordinatorNotAvailableError", "logger": "aiokafka.consumer.group_coordinator", "file": "group_coordinator.py", "function": "ensure_coordinator_known", "line": 545, "host": "ai-form-service-74664bd76c-dkcb8", "process_id": 1, "thread_id": 139896072059776, "environment": "staging", "version": "1.0.0"}
{"timestamp": "2024-11-19T18:27:18.091122Z", "level": "ERROR", "message": "Unable connect to node with id 0: ", "logger": "aiokafka", "file": "client.py", "function": "_get_conn", "line": 460, "host": "ai-form-service-74664bd76c-dkcb8", "process_id": 1, "thread_id": 139896072059776, "environment": "staging", "version": "1.0.0"}

[scholzj]

If you want to change the replication factor of any topic just because you scaled the cluster, you have to do it yourself. Strimzi will never do it for you. It might of course change the replication factor if you change it in the KafkaTopic CR, but that is your configuration change / your input.

[haroon9]

@scholzj thanks for the answer. In the configuration I do specify that offsets.topic.replication.factor: 3 but as I said it does not work for the internal __consumer_offsets topic. I understand your point that I have to change the replication factor of a topic in KafkaTopic CR and that is how I did it for all my topics.

[haroon9]

Also, does the autoRebalance feature reassign partition replicas of the __consumer_offset topic onto new brokers?

[scholzj]

@scholzj thanks for the answer. In the configuration I do specify that offsets.topic.replication.factor: 3 but as I said it does not work for the internal __consumer_offsets topic. I understand your point that I have to change the replication factor of a topic in KafkaTopic CR and that is how I did it for all my topics.

That is the Kafka configuration - so we just pass it to Kafka. I think Kafka uses it only when a new cluster starts and the topic is created for the first time. It would not be used to change the topic once it already exists.

Also, does the autoRebalance feature reassign partition replicas of the __consumer_offset topic onto new brokers?

Any kind of rebalance will reassign all topics if desired. But it will not permanently change its replication factor.