Expose information on kafka topics and consumer groups in system status page

[macobo]

Is your feature request related to a problem?

I've diagnosed multiple outages by leveraging kafka consumer group statistics, e.g. via a command like the following:

kubectl exec --stdin --tty posthog-posthog-kafka-0 -- bash -c "kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --all-groups"

This command exposes important information about the current offset and lag per partition per consumer group

Describe the solution you'd like

Expose information from that command + on kafka topics. Specifically:

• How many messages in each topic (min, max offset per partition)
• Current offset, lag per consumer group

This can then be leveraged to figure out if some service is down. E.g. clickhouse-ingestion lagging indicates plugin-server is down, and it being further behind than topic/partition min indicates data loss!

Describe alternatives you've considered

Document manual commands only.

Additional context

This might be a viable approach for exposing this data.

cc @fuziontech, @tiina303, @yakkomajuri and @guidoiaquinti for prioritization.

Thank you for your feature request – we love each and every one!

[yakkomajuri]

I have used KafkaAdminClient for stats before. Definitely displaying some metrics here would be useful

[macobo]

Took a brief dig - KafkaAdminClient is woefully underpowered for this right now (no way to get min/max offsets, lag etc) but it's possible to do this by extending the library in relatively simple ways. Related: dpkp/kafka-python#2278

[posthog-bot]

This issue hasn't seen activity in two years! If you want to keep it open, post a comment or remove the stale label – otherwise this will be closed in two weeks.

[posthog-bot]

This issue was closed due to lack of activity. Feel free to reopen if it's still relevant.