[Bug]: 0.43 in local kind cluster, fails with DNS problems

[XBeg9]

Bug Description

My connector cluster just randomly fails to connect with this warning:

WARN [Worker clientId=connect-connector-connect-0.connector-connect.default.svc:8083, groupId=connector-cluster] Error connecting to node cluster-kafka-0.cluster-kafka-brokers.default.svc:9093 (id: 2147483647 rack: null) (org.apache.kafka.clients.NetworkClient) [DistributedHerder-connect-connector-connect-0.connector-connect.default.svc:8083-1]
java.net.UnknownHostException: cluster-kafka-0.cluster-kafka-brokers.default.svc

I have equal config on my production (non-local environment, and have zero issues). Something with kind cluster, and DNS resolution.

What I have tried so far:

1. go zookeeper less (better, kafka starts, but connect still doesn't work)
2. useServiceDnsDomain: true
3. KUBERNETES_SERVICE_DNS_DOMAIN = cluster.local

Steps to reproduce

1. create local kind cluster with ctlptl create cluster kind --registry=ctlptl-registry --min-cpus=8 --kubernetes-version 1.29 (I have used >1.29, same issue)
2. helm install (default values), also used latest link.
3. apply this yaml

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: kafka
  labels:
    strimzi.io/cluster: cluster
spec:
  replicas: 1
  roles:
    - controller
    - broker
  storage:
    type: jbod
    volumes:
      - id: 0
        type: persistent-claim
        size: 10Gi
        deleteClaim: true
        kraftMetadata: shared
---
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: cluster
  annotations:
    strimzi.io/node-pools: enabled
    strimzi.io/kraft: enabled
spec:
  kafka:
    version: 3.8.0
    metadataVersion: 3.8-IV0
    resources:
      requests:
        memory: 2Gi
        cpu: 500m
      limits:
        memory: 2Gi
        cpu: "1"
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
        # configuration:
        #   useServiceDnsDomain: true
      - name: tls
        port: 9093
        type: internal
        tls: false
        authentication:
          type: scram-sha-512
      - name: external
        port: 9094
        tls: false
        type: internal
        authentication:
          type: scram-sha-512
    authorization:
      type: simple
      superUsers:
        - kafkauser
    config:
      offsets.topic.replication.factor: 1
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
      default.replication.factor: 1
      min.insync.replicas: 1
  entityOperator:
    topicOperator: {}
    userOperator: {}
---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: connector
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  image: kafkaconnect
  replicas: 1
  bootstrapServers: cluster-kafka-bootstrap:9093
  authentication:
    type: scram-sha-512
    username: kafkauser
    passwordSecret:
      secretName: kafkauser-secret
      password: password
  config:
    group.id: connector-cluster
    offset.storage.topic: connector-offsets
    config.storage.topic: connector-configs
    status.storage.topic: connector-status
    config.storage.replication.factor: -1
    offset.storage.replication.factor: -1
    status.storage.replication.factor: -1

Expected behavior

connector should connect to cluster

Strimzi version

0.43, 0.42

Kubernetes version

1.29, 1.31

Installation method

yaml, helm

Infrastructure

kind

Configuration files and logs

No response

Additional context

No response

[scholzj]

The DNS name is valid, so I think you need to investigate your DNS. One things you can try is to use other namespace than default as this namespace has in some Kubernetes distributions many special properties.

[XBeg9]

The DNS name is valid, so I think you need to investigate your DNS. One things you can try is to use other namespace than default as this namespace has in some Kubernetes distributions many special properties.

I have used kafka namespace, same issue. Any ideas where should I dig into?

[scholzj]

I do not know, sorry. I guess you could try if you can resolve it from other Pods etc. But I do not have much knowledge about the DNS components in Kubernetes beyond that, sorry.

[XBeg9]

@scholzj

I found on my side that coredns resolves only domains with cluster.local, while I have set

Can you please say from where this cluster-kafka-0.cluster-kafka-brokers.kafka.svc:9093 domain coming inside connect-cluster. Can I override it?

 WARN [debezium-postgres-connector|task-0] [Producer clientId=connector-producer-debezium-postgres-connector-0] Error connecting to node cluster-kafka-0.cluster-kafka-brokers.kafka.svc:9093 (id: 0 rack: null) (org.apache.kafka.clients.NetworkClient) [kafka-producer-network-thread | connector-producer-debezium-postgres-connector-0]
java.net.UnknownHostException: cluster-kafka-0.cluster-kafka-brokers.kafka.svc
at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:801)
at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1533)
at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1385)
at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1306)
at org.apache.kafka.clients.DefaultHostResolver.resolve(DefaultHostResolver.java:27)
at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:124)
at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.resolveAddresses(ClusterConnectionStates.java:536)
at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:511)
at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:466)
at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:173)
at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:1070)
at org.apache.kafka.clients.NetworkClient.access$800(NetworkClient.java:76)
at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1259)
at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1159)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:592)
at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:348)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:250)
at java.base/java.lang.Thread.run(Thread.java:840)

this is my connect cluster config

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: connector
  namespace: kafka
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  image: kafkaconnect
  replicas: 1
  bootstrapServers: cluster-kafka-bootstrap.kafka.svc.cluster.local:9093
  authentication:
    type: scram-sha-512
    username: kafkauser
    passwordSecret:
      secretName: kafkauser-secret
      password: password
  config:
    group.id: connector-cluster
    offset.storage.topic: connector-offsets
    config.storage.topic: connector-configs
    status.storage.topic: connector-status
    config.storage.replication.factor: -1
    offset.storage.replication.factor: -1
    status.storage.replication.factor: -1

and I use in my listeners

configuration:
          useServiceDnsDomain: true

[XBeg9]

nslookup cluster-kafka-0.cluster-kafka-brokers.kafka.svc
Server:         10.96.0.10
Address:        10.96.0.10:53

** server can't find cluster-kafka-0.cluster-kafka-brokers.kafka.svc: NXDOMAIN

/ # nslookup cluster-kafka-0.cluster-kafka-brokers.kafka.svc.cluster.local
Server:         10.96.0.10
Address:        10.96.0.10:53


Name:   cluster-kafka-0.cluster-kafka-brokers.kafka.svc.cluster.local
Address: 10.244.0.49

cat /etc/resolv.conf 
search kafka.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.96.0.10
options ndots:5

[scholzj]

You have to make sure your DNS setup can resolve the addresses without the suffix. That is how Kubernetes works. While using the useServiceDnsDomain might work around it for your own listeners, it won't work for the cluster internally.

[scholzj]

(Also, you configure useServiceDnsDomain on the listener for port 9092 but use 9093 in your Connect cluster. So that does not align.)

[XBeg9]

You have to make sure your DNS setup can resolve the addresses without the suffix. That is how Kubernetes works. While using the useServiceDnsDomain might work around it for your own listeners, it won't work for the cluster internally.

Agree with you. Switched to k3d from kind right now and it looks like working with no issues, no need for custom configs / etc. Lost a couple of days for nothing.