librdkafka consumer rd_kafka_t thread safety

[stuart-byma]

It seems that somewhere between v2.3.0 and v2.6.1 the thread safety-ness of rd_kafka_t consumer changed. In particular, I now see thread santizer errors at rdkafka_int.h:1176, basically where the rk_ts_last_poll_start and end are set. Now I see a comment in the interface code stating that these members are specifically "Not an atomic as Kafka consumer isn't thread safe".

We are using the consumer to consume messages from specific topic partitions, one topic/partition per thread, where each thread has a rd_kafka_queue_t and calls rd_kafka_consume_start_queue() to start and then polls its queue.

Now each poll operation is no longer thread safe, and they clobber the rk_ts_last_poll_start values.

Why was this changed? I thought rd_kafka_t was supposed to be thread safe?

Do I need to now create a separate rd_kafka_t consumer for each thread? Thanks in advance for any help 🙏

[stuart-byma]

This is the PR in question: #4808

Docs and other comments I have seen all indicate that rd_kafka_t is thread safe. But as of this PR being merged, that is no longer the case.