cdc-service initiates with CONECTED status and ends with RECONNECTED

[djimenez13]

Hi everyone!

I'm so excited about using transaction management among several services via eventuate. This is my first time doing this ;)
Actually, I was be able to run this project https://github.com/eventuate-tram/eventuate-tram-sagas-examples-customers-and-orders on my laptop using docker-compose (README step by step).
So, I decided to try with another project (Credit Card Example application of the Chris' Distributed Data Patterns BootCamp) using Spring Tool Suite with Maven, but the result was not good. I decided to do the debug and I realized that CDC Service could be the problem because I noticed that cdc-service container is unhealthy.

For that reason, I decide to run just 3 docker containers (zookeeper, kafka and cdc-service) via docker-compose using this config:

version: '3'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    ports:
      - 22181:2181
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  kafka:
    image: confluentinc/cp-kafka:latest
    depends_on:
      - zookeeper
    ports:
      - 29092:29092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

  cdc-service:
    image: eventuateio/eventuate-cdc-service:0.15.0.RELEASE
    ports:
      - "8099:8080"
    depends_on:
      - kafka
      - zookeeper
    environment:
      SPRING_DATASOURCE_URL: jdbc:postgresql://localhost:5432/eventuate
      SPRING_DATASOURCE_USERNAME: postgres
      SPRING_DATASOURCE_PASSWORD: admin
      SPRING_DATASOURCE_TEST_ON_BORROW: "true"
      SPRING_DATASOURCE_VALIDATION_QUERY: SELECT 1
      SPRING_DATASOURCE_DRIVER_CLASS_NAME: org.postgresql.Driver
      EVENTUATE_DATABASE_SCHEMA: eventuate
      EVENTUATELOCAL_KAFKA_BOOTSTRAP_SERVERS: kafka:29092
      EVENTUATELOCAL_ZOOKEEPER_CONNECTION_STRING: zookeeper:2181
      EVENTUATE_CDC_TYPE: EventuateTram
      EVENTUATELOCAL_CDC_SOURCE_TABLE_NAME: MESSAGE
      EVENTUATELOCAL_CDC_READER_NAME: PostgresPollingReader
      SPRING_PROFILES_ACTIVE: EventuatePolling
      EVENTUATE_OUTBOX_ID: 1
      JAVA_OPTS: -Xmx64m

..and the output is:

C:\distributed_data_labs\01\self-study-credit-card\eventuate-docker>docker-compose up -d
[+] Running 4/4
 - Network eventuate-docker_default          Created                                                                                                                                                                                    0.0s
 - Container eventuate-docker-zookeeper-1    Started                                                                                                                                                                                    1.0s
 - Container eventuate-docker-kafka-1        Started                                                                                                                                                                                    1.4s
 - Container eventuate-docker-cdc-service-1  Started                                                                                                                                                                                    1.8s

C:\distributed_data_labs\01\self-study-credit-card\eventuate-docker>docker-compose logs -f cdc-service
eventuate-docker-cdc-service-1  | #######
eventuate-docker-cdc-service-1  | #        #    #  ######  #    #   #####  #    #    ##     #####  ######
eventuate-docker-cdc-service-1  | #        #    #  #       ##   #     #    #    #   #  #      #    #
eventuate-docker-cdc-service-1  | #####    #    #  #####   # #  #     #    #    #  #    #     #    #####
eventuate-docker-cdc-service-1  | #        #    #  #       #  # #     #    #    #  ######     #    #
eventuate-docker-cdc-service-1  | #         #  #   #       #   ##     #    #    #  #    #     #    #
eventuate-docker-cdc-service-1  | #######    ##    ######  #    #     #     ####   #    #     #    ######
eventuate-docker-cdc-service-1  |
eventuate-docker-cdc-service-1  | 2023-04-18 04:02:06.704  INFO 1 --- [           main] i.e.cdc.main.EventuateCdcServiceMain     : Starting EventuateCdcServiceMain using Java 17.0.1 on 257e9d59dd67 with PID 1 (/eventuate-cdc-service-0.15.0-SNAPSHOT.jar started by root in /)
eventuate-docker-cdc-service-1  | 2023-04-18 04:02:06.711  INFO 1 --- [           main] i.e.cdc.main.EventuateCdcServiceMain     : The following 1 profile is active: "EventuatePolling"
eventuate-docker-cdc-service-1  | 2023-04-18 04:02:09.582  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8080 (http)
eventuate-docker-cdc-service-1  | 2023-04-18 04:02:09.602  INFO 1 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
eventuate-docker-cdc-service-1  | 2023-04-18 04:02:09.603  INFO 1 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet engine: [Apache Tomcat/9.0.65]
eventuate-docker-cdc-service-1  | 2023-04-18 04:02:09.725  INFO 1 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
eventuate-docker-cdc-service-1  | 2023-04-18 04:02:09.726  INFO 1 --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 2896 ms
eventuate-docker-cdc-service-1  | 2023-04-18 04:02:11.446  WARN 1 --- [           main] o.a.c.retry.ExponentialBackoffRetry      : maxRetries too large (2147483647). Pinning to 29
eventuate-docker-cdc-service-1  | 2023-04-18 04:02:11.590  INFO 1 --- [           main] o.a.c.f.imps.CuratorFrameworkImpl        : Starting
eventuate-docker-cdc-service-1  | 2023-04-18 04:02:11.600  INFO 1 --- [           main] org.apache.zookeeper.ZooKeeper           : Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
eventuate-docker-cdc-service-1  | 2023-04-18 04:02:11.601  INFO 1 --- [           main] org.apache.zookeeper.ZooKeeper           : Client environment:host.name=257e9d59dd67
eventuate-docker-cdc-service-1  | 2023-04-18 04:02:11.601  INFO 1 --- [           main] org.apache.zookeeper.ZooKeeper           : Client environment:java.version=17.0.1
eventuate-docker-cdc-service-1  | 2023-04-18 04:02:11.601  INFO 1 --- [           main] org.apache.zookeeper.ZooKeeper           : Client environment:java.vendor=Amazon.com Inc.
eventuate-docker-cdc-service-1  | 2023-04-18 04:02:11.601  INFO 1 --- [           main] org.apache.zookeeper.ZooKeeper           : Client environment:java.home=/usr/lib/jvm/java-17-amazon-corretto
eventuate-docker-cdc-service-1  | 2023-04-18 04:02:11.601  INFO 1 --- [           main] org.apache.zookeeper.ZooKeeper           : Client environment:java.class.path=eventuate-cdc-service-0.15.0-SNAPSHOT.jar
eventuate-docker-cdc-service-1  | 2023-04-18 04:02:11.601  INFO 1 --- [           main] org.apache.zookeeper.ZooKeeper           : Client environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib
eventuate-docker-cdc-service-1  | 2023-04-18 04:02:11.601  INFO 1 --- [           main] org.apache.zookeeper.ZooKeeper           : Client environment:java.io.tmpdir=/tmp
eventuate-docker-cdc-service-1  | 2023-04-18 04:02:11.601  INFO 1 --- [           main] org.apache.zookeeper.ZooKeeper           : Client environment:java.compiler=<NA>
eventuate-docker-cdc-service-1  | 2023-04-18 04:02:11.602  INFO 1 --- [           main] org.apache.zookeeper.ZooKeeper           : Client environment:os.name=Linux
eventuate-docker-cdc-service-1  | 2023-04-18 04:02:11.602  INFO 1 --- [           main] org.apache.zookeeper.ZooKeeper           : Client environment:os.arch=amd64
eventuate-docker-cdc-service-1  | 2023-04-18 04:02:11.602  INFO 1 --- [           main] org.apache.zookeeper.ZooKeeper           : Client environment:os.version=5.15.90.1-microsoft-standard-WSL2
eventuate-docker-cdc-service-1  | 2023-04-18 04:02:11.602  INFO 1 --- [           main] org.apache.zookeeper.ZooKeeper           : Client environment:user.name=root
eventuate-docker-cdc-service-1  | 2023-04-18 04:02:11.602  INFO 1 --- [           main] org.apache.zookeeper.ZooKeeper           : Client environment:user.home=/root
eventuate-docker-cdc-service-1  | 2023-04-18 04:02:11.602  INFO 1 --- [           main] org.apache.zookeeper.ZooKeeper           : Client environment:user.dir=/
eventuate-docker-cdc-service-1  | 2023-04-18 04:02:11.604  INFO 1 --- [           main] org.apache.zookeeper.ZooKeeper           : Initiating client connection, connectString=zookeeper:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@23a9ba52
eventuate-docker-cdc-service-1  | 2023-04-18 04:02:11.636  INFO 1 --- [r_default:2181)] org.apache.zookeeper.ClientCnxn          : Opening socket connection to server eventuate-docker-zookeeper-1.eventuate-docker_default/172.28.0.2:2181. Will not attempt to authenticate using SASL (unknown error)
eventuate-docker-cdc-service-1  | 2023-04-18 04:02:11.648  INFO 1 --- [r_default:2181)] org.apache.zookeeper.ClientCnxn          : Socket connection established to eventuate-docker-zookeeper-1.eventuate-docker_default/172.28.0.2:2181, initiating session
eventuate-docker-cdc-service-1  | 2023-04-18 04:02:11.692  INFO 1 --- [           main] o.a.k.clients.producer.ProducerConfig    : ProducerConfig values:
eventuate-docker-cdc-service-1  |       acks = 1
eventuate-docker-cdc-service-1  |       batch.size = 16384
eventuate-docker-cdc-service-1  |       bootstrap.servers = [kafka:29092]
eventuate-docker-cdc-service-1  |       buffer.memory = 33554432
eventuate-docker-cdc-service-1  |       client.dns.lookup = default
eventuate-docker-cdc-service-1  |       client.id =
eventuate-docker-cdc-service-1  |       compression.type = none
eventuate-docker-cdc-service-1  |       connections.max.idle.ms = 540000
eventuate-docker-cdc-service-1  |       delivery.timeout.ms = 120000
eventuate-docker-cdc-service-1  |       enable.idempotence = true
eventuate-docker-cdc-service-1  |       interceptor.classes = []
eventuate-docker-cdc-service-1  |       key.serializer = class org.apache.kafka.common.serialization.StringSerializer
eventuate-docker-cdc-service-1  |       linger.ms = 0
eventuate-docker-cdc-service-1  |       max.block.ms = 60000
eventuate-docker-cdc-service-1  |       max.in.flight.requests.per.connection = 5
eventuate-docker-cdc-service-1  |       max.request.size = 1048576
eventuate-docker-cdc-service-1  |       metadata.max.age.ms = 300000
eventuate-docker-cdc-service-1  |       metric.reporters = []
eventuate-docker-cdc-service-1  |       metrics.num.samples = 2
eventuate-docker-cdc-service-1  |       metrics.recording.level = INFO
eventuate-docker-cdc-service-1  |       metrics.sample.window.ms = 30000
eventuate-docker-cdc-service-1  |       partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
eventuate-docker-cdc-service-1  |       receive.buffer.bytes = 32768
eventuate-docker-cdc-service-1  |       reconnect.backoff.max.ms = 1000
eventuate-docker-cdc-service-1  |       reconnect.backoff.ms = 50
eventuate-docker-cdc-service-1  |       request.timeout.ms = 30000
eventuate-docker-cdc-service-1  |       retries = 2147483647
eventuate-docker-cdc-service-1  |       retry.backoff.ms = 100
eventuate-docker-cdc-service-1  |       sasl.client.callback.handler.class = null
eventuate-docker-cdc-service-1  |       sasl.jaas.config = null
eventuate-docker-cdc-service-1  |       sasl.kerberos.kinit.cmd = /usr/bin/kinit
eventuate-docker-cdc-service-1  |       sasl.kerberos.min.time.before.relogin = 60000
eventuate-docker-cdc-service-1  |       sasl.kerberos.service.name = null
eventuate-docker-cdc-service-1  |       sasl.kerberos.ticket.renew.jitter = 0.05
eventuate-docker-cdc-service-1  |       sasl.kerberos.ticket.renew.window.factor = 0.8
eventuate-docker-cdc-service-1  |       sasl.login.callback.handler.class = null
eventuate-docker-cdc-service-1  |       sasl.login.class = null
eventuate-docker-cdc-service-1  |       sasl.login.refresh.buffer.seconds = 300
eventuate-docker-cdc-service-1  |       sasl.login.refresh.min.period.seconds = 60
eventuate-docker-cdc-service-1  |       sasl.login.refresh.window.factor = 0.8
eventuate-docker-cdc-service-1  |       sasl.login.refresh.window.jitter = 0.05
eventuate-docker-cdc-service-1  |       sasl.mechanism = GSSAPI
eventuate-docker-cdc-service-1  |       security.protocol = PLAINTEXT
eventuate-docker-cdc-service-1  |       send.buffer.bytes = 131072
eventuate-docker-cdc-service-1  |       ssl.cipher.suites = null
eventuate-docker-cdc-service-1  |       ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
eventuate-docker-cdc-service-1  |       ssl.endpoint.identification.algorithm = https
eventuate-docker-cdc-service-1  |       ssl.key.password = null
eventuate-docker-cdc-service-1  |       ssl.keymanager.algorithm = SunX509
eventuate-docker-cdc-service-1  |       ssl.keystore.location = null
eventuate-docker-cdc-service-1  |       ssl.keystore.password = null
eventuate-docker-cdc-service-1  |       ssl.keystore.type = JKS
eventuate-docker-cdc-service-1  |       ssl.protocol = TLS
eventuate-docker-cdc-service-1  |       ssl.provider = null
eventuate-docker-cdc-service-1  |       ssl.secure.random.implementation = null
eventuate-docker-cdc-service-1  |       ssl.trustmanager.algorithm = PKIX
eventuate-docker-cdc-service-1  |       ssl.truststore.location = null
eventuate-docker-cdc-service-1  |       ssl.truststore.password = null
eventuate-docker-cdc-service-1  |       ssl.truststore.type = JKS
eventuate-docker-cdc-service-1  |       transaction.timeout.ms = 60000
eventuate-docker-cdc-service-1  |       transactional.id = null
eventuate-docker-cdc-service-1  |       value.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer
eventuate-docker-cdc-service-1  |
eventuate-docker-cdc-service-1  | 2023-04-18 04:02:11.702  INFO 1 --- [r_default:2181)] org.apache.zookeeper.ClientCnxn          : Session establishment complete on server eventuate-docker-zookeeper-1.eventuate-docker_default/172.28.0.2:2181, sessionid = 0x100024c6ad70000, negotiated timeout = 40000
eventuate-docker-cdc-service-1  | 2023-04-18 04:02:11.717  INFO 1 --- [ain-EventThread] o.a.c.f.state.ConnectionStateManager     : State change: CONNECTED
eventuate-docker-cdc-service-1  | 2023-04-18 04:02:11.729  INFO 1 --- [           main] o.a.k.clients.producer.KafkaProducer     : [Producer clientId=producer-1] Instantiated an idempotent producer.
eventuate-docker-cdc-service-1  | 2023-04-18 04:02:11.792  INFO 1 --- [           main] o.a.k.clients.producer.KafkaProducer     : [Producer clientId=producer-1] Overriding the default retries config to the recommended value of 2147483647 since the idempotent producer is enabled.
eventuate-docker-cdc-service-1  | 2023-04-18 04:02:11.793  INFO 1 --- [           main] o.a.k.clients.producer.KafkaProducer     : [Producer clientId=producer-1] Overriding the default acks to all since idempotence is enabled.
eventuate-docker-cdc-service-1  | 2023-04-18 04:02:11.805  INFO 1 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka version: 2.3.0
eventuate-docker-cdc-service-1  | 2023-04-18 04:02:11.806  INFO 1 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka commitId: fc1aaa116b661c8a
eventuate-docker-cdc-service-1  | 2023-04-18 04:02:11.806  INFO 1 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka startTimeMs: 1681790531803
eventuate-docker-cdc-service-1  | 2023-04-18 04:02:11.818  WARN 1 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Connection to node -1 (kafka/172.28.0.3:29092) could not be established. Broker may not be available.
eventuate-docker-cdc-service-1  | 2023-04-18 04:02:11.921  WARN 1 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Connection to node -1 (kafka/172.28.0.3:29092) could not be established. Broker may not be available.
eventuate-docker-cdc-service-1  | 2023-04-18 04:02:11.968  INFO 1 --- [           main] i.e.l.u.c.p.UnifiedCdcConfigurator       : Starting unified cdc pipelines
eventuate-docker-cdc-service-1  | 2023-04-18 04:02:11.991  INFO 1 --- [           main] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Starting...
eventuate-docker-cdc-service-1  | 2023-04-18 04:02:12.122  WARN 1 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Connection to node -1 (kafka/172.28.0.3:29092) could not be established. Broker may not be available.
eventuate-docker-cdc-service-1  | 2023-04-18 04:02:12.425  WARN 1 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Connection to node -1 (kafka/172.28.0.3:29092) could not be established. Broker may not be available.
eventuate-docker-cdc-service-1  | 2023-04-18 04:02:12.827  WARN 1 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Connection to node -1 (kafka/172.28.0.3:29092) could not be established. Broker may not be available.
eventuate-docker-cdc-service-1  | 2023-04-18 04:02:13.630  WARN 1 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Connection to node -1 (kafka/172.28.0.3:29092) could not be established. Broker may not be available.
eventuate-docker-cdc-service-1  | 2023-04-18 04:02:15.255  INFO 1 --- [ad | producer-1] org.apache.kafka.clients.Metadata        : [Producer clientId=producer-1] Cluster ID: 0eqOw4eqSwmmHpg5X6IK1A
eventuate-docker-cdc-service-1  | 2023-04-18 04:02:15.291  INFO 1 --- [ad | producer-1] o.a.k.c.p.internals.TransactionManager   : [Producer clientId=producer-1] ProducerId set to 0 with epoch 0
eventuate-docker-cdc-service-1  | 2023-04-18 04:04:08.824  INFO 1 --- [r_default:2181)] org.apache.zookeeper.ClientCnxn          : Client session timed out, have not heard from server in 37102ms for sessionid 0x100024c6ad70000, closing socket connection and attempting reconnect
eventuate-docker-cdc-service-1  | 2023-04-18 04:04:08.927  INFO 1 --- [ain-EventThread] o.a.c.f.state.ConnectionStateManager     : State change: SUSPENDED
eventuate-docker-cdc-service-1  | 2023-04-18 04:04:09.980  INFO 1 --- [r_default:2181)] org.apache.zookeeper.ClientCnxn          : Opening socket connection to server eventuate-docker-zookeeper-1.eventuate-docker_default/172.28.0.2:2181. Will not attempt to authenticate using SASL (unknown error)
eventuate-docker-cdc-service-1  | 2023-04-18 04:04:09.982  INFO 1 --- [r_default:2181)] org.apache.zookeeper.ClientCnxn          : Socket connection established to eventuate-docker-zookeeper-1.eventuate-docker_default/172.28.0.2:2181, initiating session
eventuate-docker-cdc-service-1  | 2023-04-18 04:04:09.999  INFO 1 --- [r_default:2181)] org.apache.zookeeper.ClientCnxn          : Session establishment complete on server eventuate-docker-zookeeper-1.eventuate-docker_default/172.28.0.2:2181, sessionid = 0x100024c6ad70000, negotiated timeout = 40000
eventuate-docker-cdc-service-1  | 2023-04-18 04:04:10.000  INFO 1 --- [ain-EventThread] o.a.c.f.state.ConnectionStateManager     : State change: RECONNECTED
eventuate-docker-cdc-service-1  | 2023-04-18 04:07:15.272  WARN 1 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Connection to node 1 (localhost/127.0.0.1:29092) could not be established. Broker may not be available.
eventuate-docker-cdc-service-1  | 2023-04-18 04:07:15.324  WARN 1 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Connection to node 1 (localhost/127.0.0.1:29092) could not be established. Broker may not be available.
eventuate-docker-cdc-service-1  | 2023-04-18 04:07:15.426  WARN 1 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Connection to node 1 (localhost/127.0.0.1:29092) could not be established. Broker may not be available.
eventuate-docker-cdc-service-1  | 2023-04-18 04:07:15.678  WARN 1 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Connection to node 1 (localhost/127.0.0.1:29092) could not be established. Broker may not be available.
eventuate-docker-cdc-service-1  | 2023-04-18 04:07:16.082  WARN 1 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Connection to node 1 (localhost/127.0.0.1:29092) could not be established. Broker may not be available.
eventuate-docker-cdc-service-1  | 2023-04-18 04:07:16.939  WARN 1 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Connection to node 1 (localhost/127.0.0.1:29092) could not be established. Broker may not be available.
eventuate-docker-cdc-service-1  | 2023-04-18 04:07:18.099  WARN 1 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Connection to node 1 (localhost/127.0.0.1:29092) could not be established. Broker may not be available.
eventuate-docker-cdc-service-1  | 2023-04-18 04:07:19.208  WARN 1 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Connection to node 1 (localhost/127.0.0.1:29092) could not be established. Broker may not be available.
eventuate-docker-cdc-service-1  | 2023-04-18 04:07:20.316  WARN 1 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Connection to node 1 (localhost/127.0.0.1:29092) could not be established. Broker may not be available.
eventuate-docker-cdc-service-1  | 2023-04-18 04:07:21.425  WARN 1 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Connection to node 1 (localhost/127.0.0.1:29092) could not be established. Broker may not be available.

However the broker is available as we see in the next image:

..and I saw that cdc-service is unhealthy and it is not able to "view" the kafka container and that's why I'm thinking that cdc-service has some kind of bug or something, because I can "view" the broker using https://www.kafkatool.com/ from my laptop.

Can anybody give me a hand with that? Thanks in advance!

[djimenez13]

I also tried to do the same with the 3 containers (Zookeper, Kafka an cdc-service) using eventuate project images via docker-compose using this config:

version: '3'
services:
  zookeeper:
    image: eventuateio/eventuate-zookeeper:0.16.0.RELEASE
    ports:
      - 2181:2181
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      KAFKA_HEAP_OPTS: -Xmx64m

  kafka:
    image: eventuateio/eventuate-kafka:0.16.0.RELEASE
    ports:
      - 9092:9092
    depends_on:
      - zookeeper
    environment:
      KAFKA_LISTENERS: LC://kafka:29092,LX://kafka:9092
      KAFKA_ADVERTISED_LISTENERS: LC://kafka:29092,LX://${DOCKER_HOST_IP:-localhost}:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LC:PLAINTEXT,LX:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: LC
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_HEAP_OPTS: -Xmx192m

  cdc-service:
    image: eventuateio/eventuate-cdc-service:0.15.0.RELEASE
    ports:
      - "8099:8080"
    depends_on:
      - kafka
      - zookeeper
    environment:
      SPRING_DATASOURCE_URL: jdbc:postgresql://localhost:5432/eventuate
      SPRING_DATASOURCE_USERNAME: postgres
      SPRING_DATASOURCE_PASSWORD: admin
      SPRING_DATASOURCE_TEST_ON_BORROW: "true"
      SPRING_DATASOURCE_VALIDATION_QUERY: SELECT 1
      SPRING_DATASOURCE_DRIVER_CLASS_NAME: org.postgresql.Driver
      EVENTUATE_DATABASE_SCHEMA: eventuate
      EVENTUATELOCAL_KAFKA_BOOTSTRAP_SERVERS: kafka:29092
      EVENTUATELOCAL_ZOOKEEPER_CONNECTION_STRING: zookeeper:2181
      EVENTUATE_CDC_TYPE: EventuateTram
      EVENTUATELOCAL_CDC_SOURCE_TABLE_NAME: MESSAGE
      EVENTUATELOCAL_CDC_READER_NAME: PostgresPollingReader
      SPRING_PROFILES_ACTIVE: EventuatePolling
      EVENTUATE_OUTBOX_ID: 1
      JAVA_OPTS: -Xmx64m

..the console output is different but is still unhealthy:

C:\distributed_data_labs\01\self-study-credit-card\eventuate-docker>docker-compose up -d
[+] Running 4/4
 - Network eventuate-docker_default          Created                                                                                                                                                                                    0.0s
 - Container eventuate-docker-zookeeper-1    Started                                                                                                                                                                                    1.0s
 - Container eventuate-docker-kafka-1        Started                                                                                                                                                                                    1.4s
 - Container eventuate-docker-cdc-service-1  Started                                                                                                                                                                                    1.8s

C:\distributed_data_labs\01\self-study-credit-card\eventuate-docker>docker-compose logs -f cdc-service
eventuate-docker-cdc-service-1  | #######
eventuate-docker-cdc-service-1  | #        #    #  ######  #    #   #####  #    #    ##     #####  ######
eventuate-docker-cdc-service-1  | #        #    #  #       ##   #     #    #    #   #  #      #    #
eventuate-docker-cdc-service-1  | #####    #    #  #####   # #  #     #    #    #  #    #     #    #####
eventuate-docker-cdc-service-1  | #        #    #  #       #  # #     #    #    #  ######     #    #
eventuate-docker-cdc-service-1  | #         #  #   #       #   ##     #    #    #  #    #     #    #
eventuate-docker-cdc-service-1  | #######    ##    ######  #    #     #     ####   #    #     #    ######
eventuate-docker-cdc-service-1  |
eventuate-docker-cdc-service-1  | 2023-04-18 04:44:13.246  INFO 1 --- [           main] i.e.cdc.main.EventuateCdcServiceMain     : Starting EventuateCdcServiceMain using Java 17.0.1 on 80f2f05c9c9b with PID 1 (/eventuate-cdc-service-0.15.0-SNAPSHOT.jar started by root in /)
eventuate-docker-cdc-service-1  | 2023-04-18 04:44:13.252  INFO 1 --- [           main] i.e.cdc.main.EventuateCdcServiceMain     : The following 1 profile is active: "EventuatePolling"
eventuate-docker-cdc-service-1  | 2023-04-18 04:44:15.896  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8080 (http)
eventuate-docker-cdc-service-1  | 2023-04-18 04:44:15.909  INFO 1 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
eventuate-docker-cdc-service-1  | 2023-04-18 04:44:15.910  INFO 1 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet engine: [Apache Tomcat/9.0.65]
eventuate-docker-cdc-service-1  | 2023-04-18 04:44:15.993  INFO 1 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
eventuate-docker-cdc-service-1  | 2023-04-18 04:44:15.994  INFO 1 --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 2643 ms
eventuate-docker-cdc-service-1  | 2023-04-18 04:44:17.361  WARN 1 --- [           main] o.a.c.retry.ExponentialBackoffRetry      : maxRetries too large (2147483647). Pinning to 29
eventuate-docker-cdc-service-1  | 2023-04-18 04:44:17.479  INFO 1 --- [           main] o.a.c.f.imps.CuratorFrameworkImpl        : Starting
eventuate-docker-cdc-service-1  | 2023-04-18 04:44:17.496  INFO 1 --- [           main] org.apache.zookeeper.ZooKeeper           : Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
eventuate-docker-cdc-service-1  | 2023-04-18 04:44:17.496  INFO 1 --- [           main] org.apache.zookeeper.ZooKeeper           : Client environment:host.name=80f2f05c9c9b
eventuate-docker-cdc-service-1  | 2023-04-18 04:44:17.496  INFO 1 --- [           main] org.apache.zookeeper.ZooKeeper           : Client environment:java.version=17.0.1
eventuate-docker-cdc-service-1  | 2023-04-18 04:44:17.496  INFO 1 --- [           main] org.apache.zookeeper.ZooKeeper           : Client environment:java.vendor=Amazon.com Inc.
eventuate-docker-cdc-service-1  | 2023-04-18 04:44:17.496  INFO 1 --- [           main] org.apache.zookeeper.ZooKeeper           : Client environment:java.home=/usr/lib/jvm/java-17-amazon-corretto
eventuate-docker-cdc-service-1  | 2023-04-18 04:44:17.497  INFO 1 --- [           main] org.apache.zookeeper.ZooKeeper           : Client environment:java.class.path=eventuate-cdc-service-0.15.0-SNAPSHOT.jar
eventuate-docker-cdc-service-1  | 2023-04-18 04:44:17.497  INFO 1 --- [           main] org.apache.zookeeper.ZooKeeper           : Client environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib
eventuate-docker-cdc-service-1  | 2023-04-18 04:44:17.497  INFO 1 --- [           main] org.apache.zookeeper.ZooKeeper           : Client environment:java.io.tmpdir=/tmp
eventuate-docker-cdc-service-1  | 2023-04-18 04:44:17.497  INFO 1 --- [           main] org.apache.zookeeper.ZooKeeper           : Client environment:java.compiler=<NA>
eventuate-docker-cdc-service-1  | 2023-04-18 04:44:17.497  INFO 1 --- [           main] org.apache.zookeeper.ZooKeeper           : Client environment:os.name=Linux
eventuate-docker-cdc-service-1  | 2023-04-18 04:44:17.497  INFO 1 --- [           main] org.apache.zookeeper.ZooKeeper           : Client environment:os.arch=amd64
eventuate-docker-cdc-service-1  | 2023-04-18 04:44:17.498  INFO 1 --- [           main] org.apache.zookeeper.ZooKeeper           : Client environment:os.version=5.15.90.1-microsoft-standard-WSL2
eventuate-docker-cdc-service-1  | 2023-04-18 04:44:17.498  INFO 1 --- [           main] org.apache.zookeeper.ZooKeeper           : Client environment:user.name=root
eventuate-docker-cdc-service-1  | 2023-04-18 04:44:17.498  INFO 1 --- [           main] org.apache.zookeeper.ZooKeeper           : Client environment:user.home=/root
eventuate-docker-cdc-service-1  | 2023-04-18 04:44:17.498  INFO 1 --- [           main] org.apache.zookeeper.ZooKeeper           : Client environment:user.dir=/
eventuate-docker-cdc-service-1  | 2023-04-18 04:44:17.499  INFO 1 --- [           main] org.apache.zookeeper.ZooKeeper           : Initiating client connection, connectString=zookeeper:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@23a9ba52
eventuate-docker-cdc-service-1  | 2023-04-18 04:44:17.527  INFO 1 --- [r_default:2181)] org.apache.zookeeper.ClientCnxn          : Opening socket connection to server eventuate-docker-zookeeper-1.eventuate-docker_default/172.29.0.2:2181. Will not attempt to authenticate using SASL (unknown error)
eventuate-docker-cdc-service-1  | 2023-04-18 04:44:17.538  INFO 1 --- [r_default:2181)] org.apache.zookeeper.ClientCnxn          : Socket connection established to eventuate-docker-zookeeper-1.eventuate-docker_default/172.29.0.2:2181, initiating session
eventuate-docker-cdc-service-1  | 2023-04-18 04:44:17.548  INFO 1 --- [r_default:2181)] org.apache.zookeeper.ClientCnxn          : Session establishment complete on server eventuate-docker-zookeeper-1.eventuate-docker_default/172.29.0.2:2181, sessionid = 0x1000272df700001, negotiated timeout = 40000
eventuate-docker-cdc-service-1  | 2023-04-18 04:44:17.556  INFO 1 --- [ain-EventThread] o.a.c.f.state.ConnectionStateManager     : State change: CONNECTED
eventuate-docker-cdc-service-1  | 2023-04-18 04:44:17.563  INFO 1 --- [           main] o.a.k.clients.producer.ProducerConfig    : ProducerConfig values:
eventuate-docker-cdc-service-1  |       acks = 1
eventuate-docker-cdc-service-1  |       batch.size = 16384
eventuate-docker-cdc-service-1  |       bootstrap.servers = [kafka:29092]
eventuate-docker-cdc-service-1  |       buffer.memory = 33554432
eventuate-docker-cdc-service-1  |       client.dns.lookup = default
eventuate-docker-cdc-service-1  |       client.id =
eventuate-docker-cdc-service-1  |       compression.type = none
eventuate-docker-cdc-service-1  |       connections.max.idle.ms = 540000
eventuate-docker-cdc-service-1  |       delivery.timeout.ms = 120000
eventuate-docker-cdc-service-1  |       enable.idempotence = true
eventuate-docker-cdc-service-1  |       interceptor.classes = []
eventuate-docker-cdc-service-1  |       key.serializer = class org.apache.kafka.common.serialization.StringSerializer
eventuate-docker-cdc-service-1  |       linger.ms = 0
eventuate-docker-cdc-service-1  |       max.block.ms = 60000
eventuate-docker-cdc-service-1  |       max.in.flight.requests.per.connection = 5
eventuate-docker-cdc-service-1  |       max.request.size = 1048576
eventuate-docker-cdc-service-1  |       metadata.max.age.ms = 300000
eventuate-docker-cdc-service-1  |       metric.reporters = []
eventuate-docker-cdc-service-1  |       metrics.num.samples = 2
eventuate-docker-cdc-service-1  |       metrics.recording.level = INFO
eventuate-docker-cdc-service-1  |       metrics.sample.window.ms = 30000
eventuate-docker-cdc-service-1  |       partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
eventuate-docker-cdc-service-1  |       receive.buffer.bytes = 32768
eventuate-docker-cdc-service-1  |       reconnect.backoff.max.ms = 1000
eventuate-docker-cdc-service-1  |       reconnect.backoff.ms = 50
eventuate-docker-cdc-service-1  |       request.timeout.ms = 30000
eventuate-docker-cdc-service-1  |       retries = 2147483647
eventuate-docker-cdc-service-1  |       retry.backoff.ms = 100
eventuate-docker-cdc-service-1  |       sasl.client.callback.handler.class = null
eventuate-docker-cdc-service-1  |       sasl.jaas.config = null
eventuate-docker-cdc-service-1  |       sasl.kerberos.kinit.cmd = /usr/bin/kinit
eventuate-docker-cdc-service-1  |       sasl.kerberos.min.time.before.relogin = 60000
eventuate-docker-cdc-service-1  |       sasl.kerberos.service.name = null
eventuate-docker-cdc-service-1  |       sasl.kerberos.ticket.renew.jitter = 0.05
eventuate-docker-cdc-service-1  |       sasl.kerberos.ticket.renew.window.factor = 0.8
eventuate-docker-cdc-service-1  |       sasl.login.callback.handler.class = null
eventuate-docker-cdc-service-1  |       sasl.login.class = null
eventuate-docker-cdc-service-1  |       sasl.login.refresh.buffer.seconds = 300
eventuate-docker-cdc-service-1  |       sasl.login.refresh.min.period.seconds = 60
eventuate-docker-cdc-service-1  |       sasl.login.refresh.window.factor = 0.8
eventuate-docker-cdc-service-1  |       sasl.login.refresh.window.jitter = 0.05
eventuate-docker-cdc-service-1  |       sasl.mechanism = GSSAPI
eventuate-docker-cdc-service-1  |       security.protocol = PLAINTEXT
eventuate-docker-cdc-service-1  |       send.buffer.bytes = 131072
eventuate-docker-cdc-service-1  |       ssl.cipher.suites = null
eventuate-docker-cdc-service-1  |       ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
eventuate-docker-cdc-service-1  |       ssl.endpoint.identification.algorithm = https
eventuate-docker-cdc-service-1  |       ssl.key.password = null
eventuate-docker-cdc-service-1  |       ssl.keymanager.algorithm = SunX509
eventuate-docker-cdc-service-1  |       ssl.keystore.location = null
eventuate-docker-cdc-service-1  |       ssl.keystore.password = null
eventuate-docker-cdc-service-1  |       ssl.keystore.type = JKS
eventuate-docker-cdc-service-1  |       ssl.protocol = TLS
eventuate-docker-cdc-service-1  |       ssl.provider = null
eventuate-docker-cdc-service-1  |       ssl.secure.random.implementation = null
eventuate-docker-cdc-service-1  |       ssl.trustmanager.algorithm = PKIX
eventuate-docker-cdc-service-1  |       ssl.truststore.location = null
eventuate-docker-cdc-service-1  |       ssl.truststore.password = null
eventuate-docker-cdc-service-1  |       ssl.truststore.type = JKS
eventuate-docker-cdc-service-1  |       transaction.timeout.ms = 60000
eventuate-docker-cdc-service-1  |       transactional.id = null
eventuate-docker-cdc-service-1  |       value.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer
eventuate-docker-cdc-service-1  |
eventuate-docker-cdc-service-1  | 2023-04-18 04:44:17.579  INFO 1 --- [           main] o.a.k.clients.producer.KafkaProducer     : [Producer clientId=producer-1] Instantiated an idempotent producer.
eventuate-docker-cdc-service-1  | 2023-04-18 04:44:17.625  INFO 1 --- [           main] o.a.k.clients.producer.KafkaProducer     : [Producer clientId=producer-1] Overriding the default retries config to the recommended value of 2147483647 since the idempotent producer is enabled.
eventuate-docker-cdc-service-1  | 2023-04-18 04:44:17.625  INFO 1 --- [           main] o.a.k.clients.producer.KafkaProducer     : [Producer clientId=producer-1] Overriding the default acks to all since idempotence is enabled.
eventuate-docker-cdc-service-1  | 2023-04-18 04:44:17.631  INFO 1 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka version: 2.3.0
eventuate-docker-cdc-service-1  | 2023-04-18 04:44:17.631  INFO 1 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka commitId: fc1aaa116b661c8a
eventuate-docker-cdc-service-1  | 2023-04-18 04:44:17.632  INFO 1 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka startTimeMs: 1681793057630
eventuate-docker-cdc-service-1  | 2023-04-18 04:44:17.724  INFO 1 --- [           main] i.e.l.u.c.p.UnifiedCdcConfigurator       : Starting unified cdc pipelines
eventuate-docker-cdc-service-1  | 2023-04-18 04:44:17.738  INFO 1 --- [           main] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Starting...
eventuate-docker-cdc-service-1  | 2023-04-18 04:44:17.854  INFO 1 --- [ad | producer-1] org.apache.kafka.clients.Metadata        : [Producer clientId=producer-1] Cluster ID: fqhXA5pLTlS0ALBkZVp_tg
eventuate-docker-cdc-service-1  | 2023-04-18 04:44:17.894  INFO 1 --- [ad | producer-1] o.a.k.c.p.internals.TransactionManager   : [Producer clientId=producer-1] ProducerId set to 0 with epoch 0

C:\Users\djimenez>docker ps
CONTAINER ID   IMAGE                                              COMMAND                  CREATED         STATUS                     PORTS                                        NAMES
80f2f05c9c9b   eventuateio/eventuate-cdc-service:0.15.0.RELEASE   "/bin/sh -c 'java ${…"   9 minutes ago   Up 9 minutes (unhealthy)   0.0.0.0:8099->8080/tcp                       eventuate-docker-cdc-service-1
cd549f0be43f   eventuateio/eventuate-kafka:0.16.0.RELEASE         "/bin/sh -c ./run-ka…"   9 minutes ago   Up 9 minutes (healthy)     0.0.0.0:9092->9092/tcp                       eventuate-docker-kafka-1
1bbaca7a4915   eventuateio/eventuate-zookeeper:0.16.0.RELEASE     "./run-zk.sh start-f…"   9 minutes ago   Up 9 minutes (healthy)     2888/tcp, 0.0.0.0:2181->2181/tcp, 3888/tcp   eventuate-docker-zookeeper-1

Do I have a missing configuration or something like that? Thanks in advance!

[djimenez13]

Hey everyone!. I think i did the first part. I mean, I was be able to put cdc-service healthy with the following configuration in the docker-compose:

version: '3.5'
services:
  zookeeper:
    image: eventuateio/eventuate-zookeeper:0.16.0.RELEASE
    ports:
      - 2181:2181
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      KAFKA_HEAP_OPTS: -Xmx64m

  kafka:
    image: eventuateio/eventuate-kafka:0.16.0.RELEASE
    ports:
      - 9092:9092
    depends_on:
      - zookeeper
    environment:
      KAFKA_LISTENERS: LC://kafka:29092,LX://kafka:9092
      KAFKA_ADVERTISED_LISTENERS: LC://kafka:29092,LX://${DOCKER_HOST_IP:-localhost}:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LC:PLAINTEXT,LX:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: LC
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_HEAP_OPTS: -Xmx192m

  mysql:
    image: eventuateio/eventuate-tram-sagas-mysql:0.21.0.RELEASE
    ports:
      - 3306:3306
    environment:
      - MYSQL_ROOT_PASSWORD=rootpassword
      - MYSQL_USER=mysqluser
      - MYSQL_PASSWORD=mysqlpw

  cdc-service:
    image: eventuateio/eventuate-cdc-service:0.15.0.RELEASE
    ports:
      - "8099:8080"
    depends_on:
      - mysql
      - kafka
      - zookeeper
    environment:
      SPRING_DATASOURCE_URL: jdbc:mysql://mysql/eventuate
      SPRING_DATASOURCE_USERNAME: mysqluser
      SPRING_DATASOURCE_PASSWORD: mysqlpw
      SPRING_DATASOURCE_DRIVER_CLASS_NAME: com.mysql.jdbc.Driver
      EVENTUATELOCAL_KAFKA_BOOTSTRAP_SERVERS: kafka:29092
      EVENTUATELOCAL_ZOOKEEPER_CONNECTION_STRING: zookeeper:2181
      EVENTUATELOCAL_CDC_DB_USER_NAME: root
      EVENTUATELOCAL_CDC_DB_PASSWORD: rootpassword
      EVENTUATELOCAL_CDC_READER_NAME: MySqlReader
      EVENTUATELOCAL_CDC_OFFSET_STORE_KEY: MySqlBinlog
      EVENTUATELOCAL_CDC_MYSQL_BINLOG_CLIENT_UNIQUE_ID: 1234567890
      EVENTUATELOCAL_CDC_READ_OLD_DEBEZIUM_DB_OFFSET_STORAGE_TOPIC: "false"
      EVENTUATE_OUTBOX_ID: 1
      JAVA_OPTS: -Xmx64m
      SPRING_PROFILES_ACTIVE: EventuatePolling

..and the result was:

C:\distributed_data_labs\01\self-study-credit-card\eventuate-docker>docker ps
CONTAINER ID   IMAGE                                                   COMMAND                  CREATED         STATUS                   PORTS                                        NAMES
73ef0a66d7c6   eventuateio/eventuate-cdc-service:0.15.0.RELEASE        "/bin/sh -c 'java ${…"   8 minutes ago   Up 8 minutes (healthy)   0.0.0.0:8099->8080/tcp                       eventuate-docker-cdc-service-1
3c565f96d9fd   eventuateio/eventuate-kafka:0.16.0.RELEASE              "/bin/sh -c ./run-ka…"   8 minutes ago   Up 8 minutes (healthy)   0.0.0.0:9092->9092/tcp                       eventuate-docker-kafka-1
8fb36462a2b9   eventuateio/eventuate-tram-sagas-mysql:0.21.0.RELEASE   "/entrypoint.sh mysq…"   8 minutes ago   Up 8 minutes (healthy)   0.0.0.0:3306->3306/tcp, 33060-33061/tcp      eventuate-docker-mysql-1
bae7126d2e18   eventuateio/eventuate-zookeeper:0.16.0.RELEASE          "./run-zk.sh start-f…"   8 minutes ago   Up 8 minutes (healthy)   2888/tcp, 0.0.0.0:2181->2181/tcp, 3888/tcp   eventuate-docker-zookeeper-1

The next thing I will do is run my spring boot app and test the connection with cdc-service. I mean, I will do a transaction and see if cdc-service get the data from the outbox_table and put it in the kafka broker.
Regards!