Issue with Confluent Kafka Python and MSK: "metadata information unknown"

[tuliolemes]

Hi everyone,

I hope you're all doing well. I'm facing an issue with Confluent Kafka Python and Amazon MSK, and I'm seeking some guidance.

I'm encountering an issue while trying to process a message using Confluent Kafka Python and Amazon MSK. Below is the relevant code snippet:

def delivery_report(self, err, msg):
    if err is not None:
        print("Message delivery failed: {}".format(err))
    else:
        print("Message delivered to {} [{}]".format(msg.topic(), msg.partition()))

def process_message(self, message: str):
    self.producer.poll(0)
    self.producer.produce(self.kafka_config.topic, value=message.encode("utf-8"), on_delivery=self.delivery_report)
    self.producer.flush()

Upon executing this code, I'm receiving the following error:

%7|1706584612.249|TOPIC|ip-172-20-156-228.us-east-2.compute.internal#producer-2| [thrd:app]: New local topic: 0022300646
%7|1706584612.249|TOPPARNEW|ip-172-20-156-228.us-east-2.compute.internal#producer-2| [thrd:app]: NEW 0022300646 [-1] 0x55f75a4d8240 refcnt 0x55f75a4d82d0 (at rd_kafka_topic_new0:472)
%7|1706584613.068|NOINFO|ip-172-20-156-228.us-east-2.compute.internal#producer-2| [thrd:main]: Topic 0022300646 metadata information unknown
%7|1706584613.068|NOINFO|ip-172-20-156-228.us-east-2.compute.internal#producer-2| [thrd:main]: Topic 0022300646 partition count is zero: should refresh metadata
%7|1706584613.068|CONNECT|ip-172-20-156-228.us-east-2.compute.internal#producer-2| [thrd:main]: Cluster connection already in progress: refresh unavailable topics
%7|1706584613.068|METADATA|ip-172-20-156-228.us-east-2.compute.internal#producer-2| [thrd:main]: Hinted cache of 1/1 topic(s) being queried
%7|1706584613.068|METADATA|ip-172-20-156-228.us-east-2.compute.internal#producer-2| [thrd:main]: Skipping metadata refresh of 1 topic(s): refresh unavailable topics: no usable brokers
%7|1706584613.068|CONNECT|ip-172-20-156-228.us-east-2.compute.internal#producer-2| [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 49ms: no cluster connection

My configuration is as follows:

%7|1706584609.068|INIT|ip-172-20-156-228.us-east-2.compute.internal#producer-2| [thrd:app]: librdkafka v2.3.0 (0x20300ff) ip-172-20-156-228.us-east-2.compute.internal#producer-2 initialized (builtin.features gzip,snappy,ssl,sasl,regex,lz4,sasl_plain,sasl_scram,plugins,zstd,sasl_oauthbearer,http,oidc, STRIP STATIC_LINKING GCC GXX PKGCONFIG INSTALL GNULD LIBDL PLUGINS ZLIB SSL ZSTD CURL HDRHISTOGRAM SYSLOG SNAPPY SOCKEM SASL_SCRAM SASL_OAUTHBEARER OAUTHBEARER_OIDC CRC32C_HW, debug 0xfffff)
%7|1706584609.068|CONF|ip-172-20-156-228.us-east-2.compute.internal#producer-2| [thrd:app]: Client configuration:
%7|1706584609.068|CONF|ip-172-20-156-228.us-east-2.compute.internal#producer-2| [thrd:app]:   client.id = ip-172-20-156-228.us-east-2.compute.internal
%7|1706584609.068|CONF|ip-172-20-156-228.us-east-2.compute.internal#producer-2| [thrd:app]:   metadata.broker.list = boot[...
:9098
%7|1706584609.068|CONF|ip-172-20-156-228.us-east-2.compute.internal#producer-2| [thrd:app]:   security.protocol = sasl_ssl
%7|1706584609.068|CONF|ip-172-20-156-228.us-east-2.compute.internal#producer-2| [thrd:app]:   sasl.mechanisms = OAUTHBEARER
%7|1706584609.068|CONF|ip-172-20-156-228.us-east-2.compute.internal#producer-2| [thrd:app]:   sasl.oauthbearer.config = [redacted]

I'm encountering the error "Topic metadata information unknown" and "no usable brokers." Can you provide insights into why this might be happening and how I can resolve it?

Thank you!

[tuliolemes]

I was using the config field for other kafka python library "sasl.oauthbearer.config" instead of using "oauth_cb".