This project will collect Kafka consumer lag and provide them via Prometheus.
Prometheus is a standard way to represent metrics in a modern cross-platform manner. KafkaExLagExporter exposes several metrics as an HTTP endpoint that can be readily scraped by Prometheus.
kafka_consumergroup_group_topic_sum_lag
Labels: cluster_name, group, topic, consumer_id, member_host
The sum of the difference between the last produced offset and the last consumed offset of all partitions in this topic for this group.
kafka_consumergroup_group_lag
Labels: cluster_name, group, partition, topic, member_host, consumer_id
The difference between the last produced offset and the last consumed offset for this partition in this topic partition for this group.
docker run -ti --net="host" -e KAFKA_BROKERS=redpanda:29092 -p 4000:4000 lechindianer/kafkaex_lag_exporter:0.2.0
Now you can check the exposed metrics at http://localhost:4000.
KafkaExLagExporter uses 5 seconds as default interval to update the lags. If you want to configure it to use another
value set KAFKA_EX_INTERVAL_MS
, i.e.
docker run -ti --net="host" -e KAFKA_BROKERS=redpanda:29092 -e KAFKA_EX_INTERVAL_MS=10000 -p 4000:4000 \
lechindianer/kafkaex_lag_exporter:0.2.0
To start the project locally:
KAFKA_BROKERS="localhost:9092" iex -S mix
There is also a Docker compose file included which will start Kafka, serve Kowl (Web UI for Kafka) and start KafkaexLagExporter:
docker compose up --build
Kowl is served at http://localhost:8080.
MIX_ENV=test mix test --no-start
Don't forget to check credo for code violations:
mix credo
This project also leverages the use of typespecs in order to provide static code checking:
mix dialyzer
Source is on Gitlab.
The initial project Kafka Lag Exporter was a huge inspiration for me creating my first real Elixir project. Thank you!