Proposal: flag to use tags instead of metric names to distinguish rate limit configs

[dweitzman]

Checking in here before sending out code for this.

The proposal is to add a setting like 'STATS_WITH_TAGS' that changes from reporting config stats like:

ratelimit.service.rate_limit.<domain1>.<k1>_<v1>.<k2>.total_hits
ratelimit.service.rate_limit.<domain1>.<k3>.total_hits

To something more like:

ratelimit.service.rate_limit.total_hits config=<domain1>.<k1>_<v1>.<k2>
ratelimit.service.rate_limit.total_hits config=.<domain1>.<k1>_<v1>.<k2>

The non-tagged stats would be deprecated, and on a long-term timeframe maybe the stats reporting would migrate away from the gostats library to the opentelemetry library when it's considered sufficiently mature.

Motivations:

• Some (all?...) dashboard systems make it easier to do aggregation over tags vs over metrics. For example, today both ratelimit.service.rate_limit.<config_name>.over_limit and ratelimit.service.rate_limit.<config_name>.over_limit_with_cache are metric names, and the dashboarding system I'm working with seems to have inflexible aggregration + globbing so that there doesn't seem to be a way to make a graph including the "over_limit" that doesn't also include the "over_limit_with_cache"
• statsd doesn't have an official protocol and implementations are not all compatible, so moving away from it as a default over time toward supporting protocols real standards feels sensible. Some people will also probably want to ingest into places like Prometheus where stats are more expressive than statsd and can report descriptions of what the metrics are.

[mattklein123]

statsd is still widely used, so I don't think we can deprecate it.

My recommendation would be to support both formats, much like Envoy does.

[dweitzman]

My thought for the short term is to factor out an interface for reporting stats so that all the direct dependencies on the gostats library can be isolated within one file.

Being able to replace the entire stats backend with an environment variable or by changing a single file seems useful (it'd make my life easier, at least)

[mattklein123]

Being able to replace the entire stats backend with an environment variable or by changing a single file seems useful (it'd make my life easier, at least)

Sure sounds good.