Issue when KafkaConnector #8083

dfarkash · 2023-02-09T10:51:56Z

dfarkash
Feb 9, 2023

I am trying to set up debezium on my k8s cluster and send the collected data to AWS MSK cluster.
I was able to configure the KafkaConnect, it sucsessfuly connected to the cluster and automatically created the topics that its requires, but when configuring the KafkaConnector components it failed.

Tasks: Id: 0 State: FAILED Trace: org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata

In the MSK logs, I see those logs
INFO [SocketServer listenerType=ZK_BROKER, nodeId=2] Failed authentication with /INTERNAL_IP (channelId=INTERNAL_IP-INTERNAL_IP-725) (SSL handshake failed) (org.apache.kafka.common.network.Selector)

scholzj · 2023-02-09T14:11:27Z

scholzj
Feb 9, 2023
Maintainer

Without a full log it is not really clear what error you got where. I think you should share full logs. The MSK log suggests something is trying to connect without TLS (authentication? Or just encryption?) to an interface that uses TLS. So it would be likely a configuration issue. But without seeing the logs, configs etc., hard to give anything more specific.

5 replies

dfarkash Feb 9, 2023
Author

Kafka Connector Logs:

2023-02-09 14:59:10,603 INFO [debezium-connector-sql|task-0] [Producer clientId=connector-producer-debezium-connector-sql-0] Cluster ID: pyubtf1KQ5qHBYGSOAsWng (org.apache.kafka.clients.Metadata) [kafka-producer-network-thread | connector-producer-debezium-connector-sql-0]
2023-02-09 14:59:10,802 INFO [debezium-connector-sql|task-0] [Producer clientId=core-schemahistory] Node -1 disconnected. (org.apache.kafka.clients.NetworkClient) [kafka-producer-network-thread | core-schemahistory]
2023-02-09 14:59:10,802 INFO [debezium-connector-sql|task-0] [Producer clientId=core-schemahistory] Cancelled in-flight API_VERSIONS request with correlation id 0 due to node -1 being disconnected (elapsed time since creation: 267ms, elapsed time since send: 267ms, request timeout: 30000ms) (org.apache.kafka.clients.NetworkClient) [kafka-producer-network-thread | core-schemahistory]
2023-02-09 14:59:10,802 WARN [debezium-connector-sql|task-0] [Producer clientId=core-schemahistory] Bootstrap broker b-2.qamskcls.mj6i9q.c8.kafka.eu-west-1.amazonaws.com:9096 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient) [kafka-producer-network-thread | core-schemahistory]
2023-02-09 14:59:10,848 INFO [debezium-connector-sql|task-0] [Consumer clientId=core-schemahistory, groupId=core-schemahistory] Node -2 disconnected. (org.apache.kafka.clients.NetworkClient) [task-thread-debezium-connector-sql-0]
2023-02-09 14:59:10,848 INFO [debezium-connector-sql|task-0] [Consumer clientId=core-schemahistory, groupId=core-schemahistory] Cancelled in-flight API_VERSIONS request with correlation id 1 due to node -2 being disconnected (elapsed time since creation: 312ms, elapsed time since send: 312ms, request timeout: 30000ms) (org.apache.kafka.clients.NetworkClient) [task-thread-debezium-connector-sql-0]
2023-02-09 14:59:10,848 WARN [debezium-connector-sql|task-0] [Consumer clientId=core-schemahistory, groupId=core-schemahistory] Bootstrap broker b-1.qamskcls.mj6i9q.c8.kafka.eu-west-1.amazonaws.com:9096 (id: -2 rack: null) disconnected (org.apache.kafka.clients.NetworkClient) [task-thread-debezium-connector-sql-0]
2023-02-09 14:59:10,955 INFO [debezium-connector-sql|task-0] [Producer clientId=core-schemahistory] Node -2 disconnected. (org.apache.kafka.clients.NetworkClient) [kafka-producer-network-thread | core-schemahistory]
2023-02-09 14:59:10,955 INFO [debezium-connector-sql|task-0] [Producer clientId=core-schemahistory] Cancelled in-flight API_VERSIONS request with correlation id 1 due to node -2 being disconnected (elapsed time since creation: 53ms, elapsed time since send: 53ms, request timeout: 30000ms) (org.apache.kafka.clients.NetworkClient) [kafka-producer-network-thread | core-schemahistory]
2023-02-09 14:59:10,955 WARN [debezium-connector-sql|task-0] [Producer clientId=core-schemahistory] Bootstrap broker b-1.qamskcls.mj6i9q.c8.kafka.eu-west-1.amazonaws.com:9096 (id: -2 rack: null) disconnected (org.apache.kafka.clients.NetworkClient) [kafka-producer-network-thread | core-schemahistory]

MSK Logs:

2023-02-09T16:29:43.000+02:00	[2023-02-09 14:29:43,361] INFO Metrics scheduler closed (org.apache.kafka.common.metrics.Metrics)
2023-02-09T16:29:43.000+02:00	[2023-02-09 14:29:43,361] INFO Closing reporter org.apache.kafka.common.metrics.JmxReporter (org.apache.kafka.common.metrics.Metrics)
2023-02-09T16:29:43.000+02:00	[2023-02-09 14:29:43,361] INFO Metrics reporters closed (org.apache.kafka.common.metrics.Metrics)
2023-02-09T16:29:43.000+02:00	[2023-02-09 14:29:43,362] INFO App info kafka.consumer for consumer-consumer-lag-2505 unregistered (org.apache.kafka.common.utils.AppInfoParser)
2023-02-09T16:29:45.000+02:00	[2023-02-09 14:29:44,028] INFO [SocketServer listenerType=ZK_BROKER, nodeId=1] Failed authentication with /INTERNAL_IP (channelId=INTERNAL_IP-INTERNAL_IP-1284) (SSL handshake failed) (org.apache.kafka.common.network.Selector)
2023-02-09T16:29:45.000+02:00	[2023-02-09 14:29:44,442] INFO [SocketServer listenerType=ZK_BROKER, nodeId=1] Failed authentication with /INTERNAL_IP (channelId=INTERNAL_IP-INTERNAL_IP-1284) (SSL handshake failed) (org.apache.kafka.common.network.Selector)
2023-02-09T16:29:45.000+02:00	[2023-02-09 14:29:44,472] INFO [SocketServer listenerType=ZK_BROKER, nodeId=1] Failed authentication with /INTERNAL_IP (channelId=INTERNAL_IP-INTERNAL_IP-1284) (SSL handshake failed) (org.apache.kafka.common.network.Selector)
2023-02-09T16:29:45.000+02:00	[2023-02-09 14:29:45,160] INFO [SocketServer listenerType=ZK_BROKER, nodeId=1] Failed authentication with /INTERNAL_IP (channelId=INTERNAL_IP-INTERNAL_IP-1284) (SSL handshake failed) (org.apache.kafka.common.network.Selector)
2023-02-09T16:29:45.000+02:00	[2023-02-09 14:29:45,739] INFO [SocketServer listenerType=ZK_BROKER, nodeId=1] Failed authentication with /INTERNAL_IP (channelId=INTERNAL_IP-INTERNAL_IP-1284) (SSL handshake failed) (org.apache.kafka.common.network.Selector)
2023-02-09T16:29:45.000+02:00	[2023-02-09 14:29:45,834] INFO [SocketServer listenerType=ZK_BROKER, nodeId=1] Failed authentication with /INTERNAL_IP (channelId=INTERNAL_IP-INTERNAL_IP-1285) (SSL handshake failed) (org.apache.kafka.common.network.Selector)
2023-02-09T16:29:47.000+02:00	[2023-02-09 14:29:46,501] INFO [SocketServer listenerType=ZK_BROKER, nodeId=1] Failed authentication with /INTERNAL_IP (channelId=INTERNAL_IP-INTERNAL_IP-1285) (SSL handshake failed) (org.apache.kafka.common.network.Selector)
2023-02-09T16:29:47.000+02:00	[2023-02-09 14:29:46,997] INFO [SocketServer listenerType=ZK_BROKER, nodeId=1] Failed authentication with /INTERNAL_IP (channelId=INTERNAL_IP-INTERNAL_IP-1285) (SSL handshake failed) (org.apache.kafka.common.network.Selector)
2023-02-09T16:29:47.000+02:00	[2023-02-09 14:29:47,053] INFO [SocketServer listenerType=ZK_BROKER, nodeId=1] Failed authentication with /INTERNAL_IP (channelId=INTERNAL_IP-INTERNAL_IP-1285) (SSL handshake failed) (org.apache.kafka.common.network.Selector)
2023-02-09T16:29:47.000+02:00	[2023-02-09 14:29:47,779] INFO [SocketServer listenerType=ZK_BROKER, nodeId=1] Failed authentication with /INTERNAL_IP (channelId=INTERNAL_IP-INTERNAL_IP-1285) (SSL handshake failed) (org.apache.kafka.common.network.Selector)
2023-02-09T16:29:49.000+02:00	[2023-02-09 14:29:48,168] INFO [SocketServer listenerType=ZK_BROKER, nodeId=1] Failed authentication with /INTERNAL_IP (channelId=INTERNAL_IP-INTERNAL_IP-1286) (SSL handshake failed) (org.apache.kafka.common.network.Selector)
2023-02-09T16:29:49.000+02:00	[2023-02-09 14:29:48,424] INFO [SocketServer listenerType=ZK_BROKER, nodeId=1] Failed authentication with /INTERNAL_IP (channelId=INTERNAL_IP-INTERNAL_IP-1286) (SSL handshake failed) (org.apache.kafka.common.network.Selector)
2023-02-09T16:29:49.000+02:00	[2023-02-09 14:29:48,857] INFO [SocketServer listenerType=ZK_BROKER, nodeId=1] Failed authentication with /INTERNAL_IP (channelId=INTERNAL_IP-INTERNAL_IP-1286) (SSL handshake failed) (org.apache.kafka.common.network.Selector)
2023-02-09T16:29:49.000+02:00	[2023-02-09 14:29:49,323] INFO [SocketServer listenerType=ZK_BROKER, nodeId=1] Failed authentication with /INTERNAL_IP (channelId=INTERNAL_IP-INTERNAL_IP-1286) (SSL handshake failed) (org.apache.kafka.common.network.Selector)
2023-02-09T16:29:49.000+02:00	[2023-02-09 14:29:49,739] INFO [SocketServer listenerType=ZK_BROKER, nodeId=1] Failed authentication with /INTERNAL_IP (channelId=INTERNAL_IP-INTERNAL_IP-1286) (SSL handshake failed) (org.apache.kafka.common.network.Selector)
2023-02-09T16:29:49.000+02:00	[2023-02-09 14:29:49,933] INFO [SocketServer listenerType=ZK_BROKER, nodeId=1] Failed authentication with /INTERNAL_IP (channelId=INTERNAL_IP-INTERNAL_IP-1287) (SSL handshake failed) (org.apache.kafka.common.network.Selector)
2023-02-09T16:29:51.000+02:00	[2023-02-09 14:29:50,334] INFO [SocketServer listenerType=ZK_BROKER, nodeId=1] Failed authentication with /INTERNAL_IP (channelId=INTERNAL_IP-INTERNAL_IP-1287) (SSL handshake failed) (org.apache.kafka.common.network.Selector)
2023-02-09T16:29:51.000+02:00	[2023-02-09 14:29:50,896] INFO [SocketServer listenerType=ZK_BROKER, nodeId=1] Failed authentication with /INTERNAL_IP (channelId=INTERNAL_IP-INTERNAL_IP-1287) (SSL handshake failed) (org.apache.kafka.common.network.Selector)
2023-02-09T16:29:51.000+02:00	[2023-02-09 14:29:50,999] INFO [SocketServer listenerType=ZK_BROKER, nodeId=1] Failed authentication with /INTERNAL_IP (channelId=INTERNAL_IP-INTERNAL_IP-1287) (SSL handshake failed) (org.apache.kafka.common.network.Selector)
2023-02-09T16:29:51.000+02:00	[2023-02-09 14:29:51,389] INFO [SocketServer listenerType=ZK_BROKER, nodeId=1] Failed authentication with /INTERNAL_IP (channelId=INTERNAL_IP-INTERNAL_IP-1287) (SSL handshake failed) (org.apache.kafka.common.network.Selector)
2023-02-09T16:29:53.000+02:00	[2023-02-09 14:29:52,114] INFO [SocketServer listenerType=ZK_BROKER, nodeId=1] Failed authentication with /INTERNAL_IP (channelId=INTERNAL_IP-INTERNAL_IP-1288) (SSL handshake failed) (org.apache.kafka.common.network.Selector)
2023-02-09T16:29:53.000+02:00	[2023-02-09 14:29:52,216] INFO [SocketServer listenerType=ZK_BROKER, nodeId=1] Failed authentication with /INTERNAL_IP (channelId=INTERNAL_IP-INTERNAL_IP-1288) (SSL handshake failed) (org.apache.kafka.common.network.Selector)
2023-02-09T16:29:53.000+02:00	[2023-02-09 14:29:52,668] INFO [SocketServer listenerType=ZK_BROKER, nodeId=1] Failed authentication with /INTERNAL_IP (channelId=INTERNAL_IP-INTERNAL_IP-1288) (SSL handshake failed) (org.apache.kafka.common.network.Selector)
2023-02-09T16:29:53.000+02:00	[2023-02-09 14:29:53,325] INFO [SocketServer listenerType=ZK_BROKER, nodeId=1] Failed authentication with /INTERNAL_IP (channelId=INTERNAL_IP-INTERNAL_IP-1288) (SSL handshake failed) (org.apache.kafka.common.network.Selector)
2023-02-09T16:29:53.000+02:00	[2023-02-09 14:29:53,575] INFO [SocketServer listenerType=ZK_BROKER, nodeId=1] Failed authentication with /INTERNAL_IP (channelId=INTERNAL_IP-INTERNAL_IP-1288) (SSL handshake failed) (org.apache.kafka.common.network.Selector)
2023-02-09T16:29:53.000+02:00	[2023-02-09 14:29:53,926] INFO [SocketServer listenerType=ZK_BROKER, nodeId=1] Failed authentication with /INTERNAL_IP (channelId=INTERNAL_IP-INTERNAL_IP-1289) (SSL handshake failed) (org.apache.kafka.common.network.Selector)
2023-02-09T16:29:55.000+02:00	[2023-02-09 14:29:54,697] INFO [SocketServer listenerType=ZK_BROKER, nodeId=1] Failed authentication with /INTERNAL_IP (channelId=INTERNAL_IP-INTERNAL_IP-1289) (SSL handshake failed) (org.apache.kafka.common.network.Selector)
2023-02-09T16:29:55.000+02:00	[2023-02-09 14:29:54,832] INFO [SocketServer listenerType=ZK_BROKER, nodeId=1] Failed authentication with /INTERNAL_IP (channelId=INTERNAL_IP-INTERNAL_IP-1289) (SSL handshake failed) (org.apache.kafka.common.network.Selector)
2023-02-09T16:29:55.000+02:00	[2023-02-09 14:29:55,092] INFO [SocketServer listenerType=ZK_BROKER, nodeId=1] Failed authentication with /INTERNAL_IP (channelId=INTERNAL_IP-INTERNAL_IP-1289) (SSL handshake failed) (org.apache.kafka.common.network.Selector)
2023-02-09T16:29:55.000+02:00	[2023-02-09 14:29:55,902] INFO [SocketServer listenerType=ZK_BROKER, nodeId=1] Failed authentication with /INTERNAL_IP (channelId=INTERNAL_IP-INTERNAL_IP-1289) (SSL handshake failed) (org.apache.kafka.common.network.Selector)
2023-02-09T16:29:57.000+02:00	[2023-02-09 14:29:56,040] INFO [SocketServer listenerType=ZK_BROKER, nodeId=1] Failed authentication with /INTERNAL_IP (channelId=INTERNAL_IP-INTERNAL_IP-1290) (SSL handshake failed) (org.apache.kafka.common.network.Selector)
2023-02-09T16:29:57.000+02:00	[2023-02-09 14:29:56,484] INFO [SocketServer listenerType=ZK_BROKER, nodeId=1] Failed authentication with /INTERNAL_IP (channelId=INTERNAL_IP-INTERNAL_IP-1290) (SSL handshake failed) (org.apache.kafka.common.network.Selector)
2023-02-09T16:29:57.000+02:00	[2023-02-09 14:29:57,163] INFO [SocketServer listenerType=ZK_BROKER, nodeId=1] Failed authentication with /INTERNAL_IP (channelId=INTERNAL_IP-INTERNAL_IP-1290) (SSL handshake failed) (org.apache.kafka.common.network.Selector)
2023-02-09T16:29:57.000+02:00	[2023-02-09 14:29:57,287] INFO [SocketServer listenerType=ZK_BROKER, nodeId=1] Failed authentication with /INTERNAL_IP (channelId=INTERNAL_IP-INTERNAL_IP-1290) (SSL handshake failed) (org.apache.kafka.common.network.Selector)
2023-02-09T16:29:57.000+02:00	[2023-02-09 14:29:57,294] INFO Retrieving credential for user: ****** [INTERNAL]

The KafkaConnect was able to connect but the KafkaConnector wasn't.
I'm trying to understand what am I missing, do I need to configure the SASL_SSL login in the KafkaConnector or it should use the KafkaConnect.

KafkaConnector:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
  name: debezium-connector-sql
  namespace: debezium
  labels:
    strimzi.io/cluster: debezium-connect-cluster
spec:
  class: io.debezium.connector.sqlserver.SqlServerConnector
  tasksMax: 1
  autoRestart:
    enabled: true
  config:
    database.hostname: db_host
    database.encrypt: false
    database.port: 1433
    database.user: db_user
    database.password: <db_password>
    database.names: <db>
    table.include.list: <db_tables>
    topic.prefix: sql
    schema.history.internal.kafka.bootstrap.servers: <msk_SASL/SCRAM_endpoint>
    schema.history.internal.kafka.topic: ge-core-debezium-orders

KafkaConnect:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: debezium-connect-cluster
  namespace: debezium
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  version: 3.3.1
  replicas: 1
  image: <image>
  bootstrapServers: <msk_SASL/SCRAM_endpoint>
  config:
    config.providers: secrets
    config.providers.secrets.class: io.strimzi.kafka.KubernetesSecretConfigProvider
    group.id: connect-cluster
    offset.storage.topic: connect-cluster-offsets
    config.storage.topic: connect-cluster-configs
    status.storage.topic: connect-cluster-status
    config.storage.replication.factor: -1
    offset.storage.replication.factor: -1
    status.storage.replication.factor: -1
  tls:
    trustedCertificates: [ ]
  authentication:
    type: scram-sha-512
    username: <user>
    passwordSecret:
      password: password
      secretName: <secretName>

scholzj Feb 9, 2023
Maintainer

You need to share the full logs ... at least the Connect one.

dfarkash Feb 9, 2023
Author

KafkaConnect_logs.log

scholzj Feb 9, 2023
Maintainer

Maybe you need to properly configure the Debezium history connection? Debezium is a bit special and has this special connection it opens on its own which needs a separate coniguration by the user. Because it tries to open many connections with security.protocol = PLAINTEXT which would cause the issue you have when connecting to TLS enabled cluster.

gutpapr Dec 3, 2024

How this was resolved?

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Strimzi

Issue when KafkaConnector #8083

{{title}}

Replies: 1 comment 5 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Strimzi

Issue when KafkaConnector #8083

dfarkash Feb 9, 2023

Replies: 1 comment · 5 replies

scholzj Feb 9, 2023 Maintainer

dfarkash Feb 9, 2023 Author

scholzj Feb 9, 2023 Maintainer

dfarkash Feb 9, 2023 Author

scholzj Feb 9, 2023 Maintainer

gutpapr Dec 3, 2024

dfarkash
Feb 9, 2023

Replies: 1 comment 5 replies

scholzj
Feb 9, 2023
Maintainer

dfarkash Feb 9, 2023
Author

scholzj Feb 9, 2023
Maintainer

dfarkash Feb 9, 2023
Author

scholzj Feb 9, 2023
Maintainer