Pull Exporter vs Exponential Bucket Histograms

[bbdouglas]

HI! I am looking into whether it is possible (or foreseeable) to be able to use a pull mode exporter with Exponential Bucket Histograms.

Right now, it seems that you need to pick one or the other. You can use OTLP with the full set of metric types, but that is push mode. Prometheus is pull-based, but the spec only mentions the text format, which can't be used to encode Exponential Bucket Histograms. Exponential Bucket Histograms can be represented as Prometheus Native Histograms, but it requires the binary (protobuf) format.

The reason I am interested in pull mode exporters is that we have found their ergonomics better suited to our use case of monitoring HTTP servers. We already have libraries and tooling for standing up and querying HTTP endpoints, and it would fit nicely into that model to also be able to scrape metrics from yet another endpoint. But Exponential Bucket Histograms are a big enough game-changer to make it a hard requirement of any telemetry system we use, which is how we find ourselves in this bind.

I'm curious if there are others who are facing this same trade-off and whether there are any plans to support Exponential Bucket Histograms in a pull mode metric exporter.

[jack-berg]

Couple of thoughts:

• The prometheus compatibility document does define how to convert exponential histograms to the prometheus format - we just haven't gotten around to implementing it yet. If its important to you we're open to contributions 🙂
• Pull based metric readers are a generic concept and PrometheusHttpServer is just one implementation. You can implement your own if you'd like!

[bbdouglas]

Thanks for the feedback, Jack. So it sounds like Prometheus is the best bet at this point. In order to get that working end-to-end with a metrics backend within the otel ecosystem, both the java client and the otel-collector would need to add support for Prometheus binary. Since the Prometheus compatibility spec is marked as Experimental, and since the Prometheus binary protobuf format itself is still experimental as well, I wonder if it might be a little early to start going down that road.

However, beginning with Prometheus v2.40, there is experimental support for native histograms, which – at least in its initial experimental state – utilizes the old Protobuf format (with some newer additions) again.... This support is experimental and might get removed again.

I was also just curious if you knew of any initiatives within the community to introduce any otel-native pull exporters. I know that we can use the MetricReader interface to implement our own, but when looking at the whole metrics pipeline end-to-end, it would be great to stick to standards-compliant solutions for compatibility and support down the whole chain.