resume and pause in KafkaConsumer (reactivex) are not working as expected and causing rebalancing of consumer group

[yogs95]

Questions

resume and pause in KafkaConsumer (reactivex) are not working as expected and causing rebalancing of consumer group

Version

3.6.0

Context

It is causing rebalancing of consumer when pause interval exceeds max.poll.interval.ms as per the documentation of client pause should not cause the rebalance

Do you have a reproducer?

Steps:

1. set max.poll.interval.ms to 10 seconds
2. pause the consumer using KafkaConsumer.pause more than 10 seconds

Extra

OS: Catelina (10.15)
JAVA: 1.8
vertx-kafka-client: 3.6.0

[aesteve]

Yes, that's extremely confusing.
pause() and resume act on the ReadStream (that's a Vert.x notion, beyond Kafka).
pause(TopicPartitions) actually calls Kafka's pause method.

If we pause() the stream, it will stop polling and be considered dead by the consumer group leader, leading to a rebalance as you mentioned.

I think we have two problems:

• The fact the methods have the name appears to the reader as a unique method overloaded (eg: is pause() a convenience method for pause(allTopicPartitionsISubscribedTo)? which could make sense
• the Javadoc doesn't really help with this confusion :(

The pause() and resume() provides global control over reading the records from the consumer.

Re-reading it now, I understand what this means, but I think we either:

• should keep a list of TopicPartitions metadata the consumer has subscribed to, and make pause() pause the stream AND pause(allSubscriptions) (but this might be a breaking change)
• or document (in bold) that pause() provides control over the flow of records in the stream and doesn't do anything Kafka-protocol related, not to be confused with pause(TopicPartition)

@vietj what do you think?

[aesteve]

See: #53