Simplification of consumer internals

[LMnet]

From here:

At the current moment, consumer internals are pretty complex: we have a KafkaConsumer itself, a hand-maded KafkaConsumerActor, a queue for messages. Also, there is some complexity with different commit methods. And most of this complexity is to make all calls to a java consumer linearized. But actually, to achieve this we could use a simple Semaphore(1). Under the hood, this semaphore uses a queue (like an actor), but all low-level queue-related machinery is hidden. With this, we could dramatically reduce the complexity of library internals and make it more contributor-friendly.

[bplommer]

That sounds good - the only thing I wonder is how we would handle prioritising client requests over polls? I wonder if instead of switching to cats.effect.Semaphore we should instead/first replace the different kind of actor requests with a single semaphore-like permit request

[bplommer]

I'm thinking something like #906 - does that fit with what you had in mind?

[aartigao]

I agree with @LMnet that current code is not contributor-friendly. That's my experience at least 😅 I find it difficult to follow and I'm pretty confident that we can simplify it by removing completely KafkaConsumerActor.

I'm working on a POC to simplify the internals (let's see how far I can get).

About how to prioritize client requests over polls: wouldn't Priority Queue be the answer to this?