Add new metric for kafka topic partition size.

[xxm404]

No description provided.

[xxm404]

New metric

kafka_topic_partition_size{partition="30",topic="__consumer_offsets"} 2443
kafka_topic_partition_size{partition="31",topic="__consumer_offsets"} 1031
kafka_topic_partition_size{partition="32",topic="__consumer_offsets"} 1738
kafka_topic_partition_size{partition="33",topic="__consumer_offsets"} 0
kafka_topic_partition_size{partition="34",topic="__consumer_offsets"} 0
kafka_topic_partition_size{partition="35",topic="__consumer_offsets"} 614
kafka_topic_partition_size{partition="36",topic="__consumer_offsets"} 578
kafka_topic_partition_size{partition="37",topic="__consumer_offsets"} 24920
kafka_topic_partition_size{partition="38",topic="__consumer_offsets"} 17611
kafka_topic_partition_size{partition="39",topic="__consumer_offsets"} 9930

[xxm404]

@danielqsj hi, I add a new metric for a topic partition size, like this

kafka_topic_partition_size{partition="30",topic="__consumer_offsets"} 2443
kafka_topic_partition_size{partition="31",topic="__consumer_offsets"} 1031
kafka_topic_partition_size{partition="32",topic="__consumer_offsets"} 1738
kafka_topic_partition_size{partition="33",topic="__consumer_offsets"} 0
kafka_topic_partition_size{partition="34",topic="__consumer_offsets"} 0
kafka_topic_partition_size{partition="35",topic="__consumer_offsets"} 614
kafka_topic_partition_size{partition="36",topic="__consumer_offsets"} 578
kafka_topic_partition_size{partition="37",topic="__consumer_offsets"} 24920
kafka_topic_partition_size{partition="38",topic="__consumer_offsets"} 17611
kafka_topic_partition_size{partition="39",topic="__consumer_offsets"} 9930

please review this PR

[atrbgithub]

This looks great 👍

[atrbgithub]

@xxm404 I've had a look at building on your code to make things a little faster as it was making a call for every partition, which was taking quite some time on a cluster with many topics/partitions.

Please see here

This builds a map of all the brokers and topics, with an array for each partition they own.

It then makes a single call for each broker with an array of topics and partitions which that broker is the leader for:

https://github.com/atrbgithub/kafka_exporter/blob/5923381978656a6fbeb91fb069c14f44ecb5f73e/kafka_exporter.go#L607

Not sure if you want to modify your PR to do something similar?

[soloradish]

It's better to follow the metric's naming convention.
Add a unit as this metric's suffix like kafka_topic_partition_size_bytes.

But I couldn't managed to find the right unit name in Kafka java client docs. Maybe you guys can figure it out ^-^

@xxm404 @atrbgithub

[turbotankist]

Any updates?

[soloradish]

Any updates?

You can use my build, https://github.com/soloradish/kafka_exporter

It expose the kafka_topic_partition_size_bytes metric. but be ware I remove all consumer related metrics, because I use kafka lag exporter for these.