Should we use OpenTelemetry or OpenMetrics semantic conventions?

[emschwartz]

Unfortunately, OpenTelemetry and OpenMetrics standardized metrics in slightly different ways such that we either need to pick one or make the queries we write configurable.

Differences between OpenTelemetry and OpenMetrics

As far as I can tell, the main differences between the two standards that are relevant to autometrics are as follows:

Separators

OpenTelemetry uses . as separators in metric names while OpenMetrics uses _.

In the Rust implementation we currently handle this difference by using the . when producing metrics with the opentelemetry crate, using _ when producing them with the prometheus crate, and replacing all .'s with _'s when writing Prometheus queries. I'm not sure this is ideal but it kind of works.

Counter suffix

OpenTelemetry: .count

OpenTelemetry's semantic conventions for metrics says:

Metric names SHOULD NOT be pluralized, unless the value being recorded represents discrete instances of a countable quantity. Generally, the name SHOULD be pluralized only if the unit of the metric in question is a non-unit (like {faults} or {operations}).

If the value being recorded represents the count of concepts signified by the namespace then the metric should be named count (within its namespace). The pluralization rule does not apply in this case.
For example if we have a namespace system.processes which contains all metrics related to the processes then to represent the count of the processes we can have a metric named system.processes.count. The suffix count here indicates that it is the count of system.processes.

This suggests that our counter metric name should be function.calls.count.

(Update: I asked why OpenTelemetry used the count suffix here but haven't received an answer yet.)

OpenMetrics: _total

OpenMetrics stipulates:

Suffixes for the respective types are:
Counter: '_total', '_created'

And it seems that at least some Prometheus libraries enforce this convention by automatically appending a _total to the end of any counter name.

Following the OpenMetrics conventions, our counter metric name should be function_calls_total.

Latency Units

OpenTelemetry: milliseconds (update: but maybe changing to seconds)

The semantic conventions do not seem to mandate that durations be tracked in milliseconds but the semantic conventions for HTTP Metrics and others use milliseconds.

There is an open discussion in open-telemetry/opentelemetry-specification#2977 about seconds vs milliseconds and how it would be a breaking change for them to switch.

Update: This comment says:

We've discussed this during the Feb. 14th specification SIG meeting:

• We will make the change to use seconds instead of ms, which aligns with Prometheus.
• The change might introduce some issue to existing specification, for example Semantic Conventions: default to seconds for duration units open-telemetry/opentelemetry-specification#2977 (comment). We believe this issue can be resolved via Hint API, and jack-berg is willing to propose a small addition to the spec (minimal Hint API for explicit histogram buckets).

OpenMetrics: seconds

OpenMetrics says:

Units and Base Units
For consistency across systems and to avoid confusion, units are largely based on SI base units. Base units include seconds, bytes, joules, grams, meters, ratios, volts, amperes, and celsius. Units should be provided where they are applicable.

---

What should we do?

Overall, my impression is that the OpenTelemetry project has more traction. However, most users of OpenTelemetry seem to be focused on tracing and the metrics specifications are listed as "experimental"...

[gagbo]

Isn't it possible to handle both cases based on a switch? As far as I understand, as a user of Autometrics, I'm going to target either an OpenMetrics setup or an OpenTelemetry setup when I start publishing metrics. Therefore some kind of global flag could be used in the library to tell which implementation (OT/OM) we'd want and generate only the metric names/units that match it

[IvanMerrill]

I would go with OTEL. It's got a lot of traction with many different vendors and is being actively worked on. The metrics specification is stable.

• OTEL started out with the intention of being a unified monitoring standard for all signal types, and specifically aimed to bring together OpenMetrics and OpenCensus to one standard
• It's well on its way to becoming the de facto standard
• It gives people the ability to use autometrics with any backend, OpenMetrics mostly Prometheus specific. The OTEL collector gives a lot of flexibility
• The amount of traction OTEL is getting means it's likely to continue to be developed longer term. I'm not sure how much OpenMetrics is being worked on any more
• It gives people the opportunity to grow their monitoring as their service matures with tracing and logs

[arendjr]

Just to add to what @IvanMerrill wrote, we are also focusing on OpenTelemetry for our logging functionality, which is reflected for instance in the ability to add OTEL metadata to our Event type. So this is another vote for OpenTelemetry.

[gouthamve]

There is an option to expose Prometheus endpoint when using OpenTelemetry SDKs.

Also, we are working on removing the . limitation in Prometheus. It won't happen immediately but it is on its way :) https://groups.google.com/g/prometheus-developers/c/-O23C4_9Ijc

[emschwartz]

Ah, excellent! That's good to know

[gouthamve]

There is a slight advantage of using OpenTelemetry SDKs is that you will support push out of the box, and not just pull.

[arendjr]

So earlier I also suggested it would be preferred to go the OTel route, but for pragmatic reasons I no longer think we should focus too much on it. We use Prometheus clients for pushing metrics and they are getting in the way of doing full OTel compatibility, because they seem to largely embrace OpenMetrics instead.

In addition, I don't think we should wait and hope for this situation to resolve itself, since Prometheus storage plays a central part in our strategy and the lack of a standard from our side is already leading to divergence between our implementations. This effectively leads us to:

• The Rust implementation using a metric named function_calls_count which is neither valid OTel (due to the underscores) nor valid OpenMetrics (due to the lacking _total suffix).
• The Go implementation trying to follow the Rust one, but using gauges to do so, thereby going against the explicit recommendations from Prometheus itself. (Even if it seems to work so far...)
• The Python and TypeScript implementations both using function_calls_count_total, thereby effectively following OpenMetrics, because that's what their Prometheus clients want them to use.
• It also leads us to require unnecessarily complex queries to cover the divergence between the above implementations.

To resolve this situation, I would suggest embracing OpenMetrics and to effectively follow the Python and TypeScript implementations. This would seem relatively easy to accomplish for both the Rust and the Go implementations (whereas the other way around is both a lot more work, and would also settle them on a metric name that follows neither standard).

Note this does not mean I would suggest giving up on OTel altogether. I think it's a nice feature of the Rust implementation that it allows OTel exports as well, which I think other clients should strive to implement too. But for Prometheus exports specifically, I think OpenMetrics is the way to go.

[emschwartz]

@gouthamve do you have an opinion on calling the metric function.calls.total? I asked a question on the OTel repo about why it recommends using count instead of total (opentelemetry-specification/discussions/3304) but didn't get a response. I wonder if you have any more insight into that.

[gouthamve]

This is the first time I am seeing this. Funnily enough, this is not adopted imo. For example, the http semantic conventions are being finalised right now, and the metric is still called http.server.active_requests: open-telemetry/opentelemetry-specification#3451

[emschwartz]

Interesting. Given that, I opened an issue to propose changing it to total here: open-telemetry/opentelemetry-specification#3457

[emschwartz]

Based on the discussion in that thread, it seems like the OTel folks are okay with getting rid of the recommendation to use count.

That will leave us with the option of replacing the word count with total or getting rid of the suffix entirely (if open-telemetry/opentelemetry-specification#3477 is accepted) and just letting the Prometheus exporter append the suffix. I think it's better to switch to the word total so the metrics we're producing more closely match what we're querying from Prometheus -- and it doesn't seem like OTel will have an issue with a metric having a .total suffix.

[emschwartz]

Alright, this new PR suggests that we should leave off the total suffix: open-telemetry/opentelemetry-specification#3493

[gouthamve]

FYI, it is part of the Spec in OTel to Prometheus that counters should have a _total appended. Any clients not doing it are a bug.

The collector will now append the units and _total from v0.76.0 (releasing next week) and once that happens, the OTel SDKs will follow suit. If you open a PR to add a _total to an SDK, please tag me and I can explain the reasoning in that PR if required.

[emschwartz]

Good to know.

Is there another spec for that behavior aside from this document? The _total suffix isn't currently mentioned there.

[gouthamve]

https://github.com/open-telemetry/opentelemetry-specification/blob/b5c6a6dc48752a3ea740f08bed0c2975bae19673/specification/compatibility/prometheus_and_openmetrics.md#sums

[gagbo]

Currently the unit appending in the Go exporter is optional

Does the mentioned change means that it will become mandatory? That probably means we need to also change this in our current Prometheus implementation if we want to have consistent metric names (Currently the Rust implementation doesn't add the _seconds suffix to the call duration metric, so no other implementation is adding it either to have alerting rules work fine)

[gouthamve]

In the future, it is likely that it will become mandatory. If I understand correctly, the default today is still to append the unit right?

[gagbo]

Yes, the default is to append the unit, and the exporter API has a WithoutUnits method

[emschwartz]

@gagbo brought up a good point that Prometheus/OpenMetrics naming conventions also include the units in the metric names. This suggests that the name of our histogram, at least as far as Prometheus is concerned, should be function_calls_duration_seconds.

OpenTelemetry says:

Units

Conventional metrics or metrics that have their units included in OpenTelemetry metadata (e.g. metric.WithUnit in Go) SHOULD NOT include the units in the metric name. Units may be included when it provides additional meaning to the metric name. Metrics MUST, above all, be understandable and usable.

This suggests that the metric can continue to be called function.calls.duration and we should ensure that it's converted to function_calls_duration_seconds when exported to Prometheus and in the generated queries.

[emschwartz]

Luckily for us, OpenTelemetry changed some of their recommendations in such a way that it's easier for us to conform to both OpenTelemetry and OpenMetrics/Prometheus naming conventions! 🎉

Here are the conclusions:

• Our counter is called function.calls and is exported to Prometheus as function_call_total
• Our histogram is called function.calls.duration and is exported to Prometheus as function_calls_duration_seconds
• Our histogram uses seconds as the units

We're going to support the old values for some time in the shared resources like dashboards and alerting rules by using regexes to match the names, but eventually we'll remove the regexes and just use these names.

The Autometrics spec was updated with these changes in #60