Alertmanager: Support UTF-8 #6898

grobinson-grafana · 2023-12-12T10:59:43Z

What this PR does

This pull request adds support for UTF-8 in the Mimir Alertmanager (please see prometheus/alertmanager#3486).

Unlike the Prometheus Alertmanager, which operates both parsers at the same time using a fallback mechanism, the Mimir Alertmanager instead operates in one of two modes: classic mode using the pkg/labels parser, and strict UTF-8 mode using the matchers/parse parser. It uses classic mode as the default mode.

To help transition a Mimir installation from classic mode to strict UTF-8 mode, the Mimir Alertmanager contains additional code that runs all API requests and configurations through the fallback parser, incrementing metrics and emitting logs should it encounter configurations that are not forwards compatible or cause disagreement.

The operator of a Mimir cluster is expected to use these metrics and logs to observe incompatible configurations, or configurations that cause disagreement, and fix them. Once all configurations have been fixed the operator can then migrate the installation to strict UTF-8 mode. From there on, the pkg/labels parser will no longer be used, and it will no longer be possible to have configurations that are incompatible with the matchers/parse parser unless the mode be reverted.

A Mimir installation can be switched to the strict UTF-8 mode using the alertmanager.utf8-strict-mode flag.

Dashboards

Here are a couple of screenshots from the dashboards that we will use to manage the rollout of UTF-8 in Mimir.

The first two screenshots show an ideal cell. There are no configurations with incompatibilities or disagreement. This cluster can be migrated to UTF-8 strict mode.

The next couple screenshots show a cell that has both disagreement and incompatibilities. The metrics in this screenshot are exaggerated as Mimir reloads configurations at regular intervals.

However, in the per tenant dashboard, we can see disagreement and incompatibilities deduplicated by tenant. In this case, we will fix the incompatible configuration but no the disagreement, as the disagreement in this screenshot is 🙂 encoded as a byte sequence.

Fixes #

Checklist

Tests updated.
Documentation added.
CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
about-versioning.md updated with experimental features.

CLAassistant · 2023-12-12T10:59:50Z

All committers have signed the CLA.

gotjosh

Code-wise this LGTM, but I reckcon we need a few integrations tests:

Integration tests that asserts on the metrics you're mentioning with the various scenarios: no disagreement, some disagreement and all but disagreement of the various objects.
Integration tests that assert on matchers that are now accepted that wouldn't be accepted before.

cmd/mimir/config-descriptor.json

gotjosh

LGTM

gotjosh · 2024-01-25T09:45:18Z

CHANGELOG.md

@@ -4,6 +4,7 @@

 ### Grafana Mimir

+* [CHANGE] Alertmanager: CLI flag `-utf8-strict-mode` has been added, and its default value is set to `true`. #6898


Suggested change

* [CHANGE] Alertmanager: CLI flag `-utf8-strict-mode` has been added, and its default value is set to `true`. #6898

* [FEATURE] Alertmanager: Added `-alertmanager.utf8-strict-mode` to control support for any UTF-8 character as part of Alertmanager configuration/API matchers and labels. It's default value is set to `false`. #6898

I think this is a feature more than a change, we tend to use whole flag and the default value is actually false.

Good catch on the true/false. I copied another CHANGELOG as an example and forgot to change this.

gotjosh · 2024-01-25T09:53:56Z

pkg/alertmanager/multitenant.go

+	// Enable UTF-8 strict mode. This means Alertmanager uses the matchers/parse parser
+	// to parse configurations and API requests, instead of pkg/labels. Use this mode
+	// once you are confident that your configuration is forwards compatible.
+	UTF8StrictMode bool `yaml:"utf8_strict_mode" category:"advanced"`


Marking this flag as advanced - means it'll need to follow the regular deprecation flow of 2 minor versions. So, you'll need at least three versions to remove it. Marking it as experimental, though, will allow you to remove it whenever you want.

See config docs for more.

I'm OK with this as is as I think there's value in making it permanent for a few versions to enable folks do to do their own migrations but thought it was worth mentioning.

TIL. Let's just make it experimental. Even if it is experimental, doesn't mean we can't follow the regular deprecation flow if we so choose?

gotjosh · 2024-01-25T09:55:17Z

pkg/mimir/modules.go

+	mode := featurecontrol.FeatureClassicMode
+	if t.Cfg.Alertmanager.UTF8StrictMode {
+		level.Debug(util_log.Logger).Log("msg", "Starting Alertmanager in UTF-8 strict mode")
+		mode = featurecontrol.FeatureUTF8StrictMode
+	} else {
+		level.Debug(util_log.Logger).Log("msg", "Starting Alertmanager in classic mode")
+	}
+	features, err := featurecontrol.NewFlags(util_log.Logger, mode)


Nit, this is pretty important and only logged once - I'd consider changing his to info.

Suggested change

mode := featurecontrol.FeatureClassicMode

if t.Cfg.Alertmanager.UTF8StrictMode {

level.Debug(util_log.Logger).Log("msg", "Starting Alertmanager in UTF-8 strict mode")

mode = featurecontrol.FeatureUTF8StrictMode

} else {

level.Debug(util_log.Logger).Log("msg", "Starting Alertmanager in classic mode")

}

features, err := featurecontrol.NewFlags(util_log.Logger, mode)

mode := featurecontrol.FeatureClassicMode

if t.Cfg.Alertmanager.UTF8StrictMode {

level.Info(util_log.Logger).Log("msg", "Starting Alertmanager in UTF-8 strict mode")

mode = featurecontrol.FeatureUTF8StrictMode

} else {

level.Info(util_log.Logger).Log("msg", "Starting Alertmanager in classic mode")

}

features, err := featurecontrol.NewFlags(util_log.Logger, mode)

gotjosh · 2024-01-25T09:56:13Z

integration/alertmanager_test.go

Thank you for adding the tests, they look beautiful!

grobinson-grafana · 2024-01-25T11:13:22Z

I had to force push to fix conflicts in CHANGELOG.md.

gotjosh

LGTM

…ollowup Some follow up from #6898 - Move the changelog entry to the right place - Add documentation about experimental features on `about-versioning` - New boolean flags tend to be suffixed with the word "enabled" if it makes sense - Minor fix to the flag documentation to aling with the changelog

grobinson-grafana force-pushed the grobinson/utf8-mimir branch from 8b10457 to 1161178 Compare January 5, 2024 14:00

grobinson-grafana changed the title ~~WIP: UTF-8 in Mimir~~ UTF-8 in Mimir Jan 5, 2024

grobinson-grafana force-pushed the grobinson/utf8-mimir branch 2 times, most recently from 804f02c to 36c8886 Compare January 11, 2024 10:11

grobinson-grafana force-pushed the grobinson/utf8-mimir branch 5 times, most recently from b3327a5 to 2767767 Compare January 19, 2024 13:00

grobinson-grafana mentioned this pull request Jan 19, 2024

Alertmanager: Update to latest main #7103

Merged

4 tasks

grobinson-grafana force-pushed the grobinson/utf8-mimir branch 12 times, most recently from d84aeb0 to b9d47e9 Compare January 24, 2024 14:14

grobinson-grafana requested a review from gotjosh January 24, 2024 14:19

grobinson-grafana self-assigned this Jan 24, 2024

grobinson-grafana force-pushed the grobinson/utf8-mimir branch from b9d47e9 to a82c4ea Compare January 24, 2024 14:23

gotjosh reviewed Jan 24, 2024

View reviewed changes

cmd/mimir/config-descriptor.json Show resolved Hide resolved

gotjosh approved these changes Jan 25, 2024

View reviewed changes

grobinson-grafana added 2 commits January 25, 2024 11:12

UTF-8 in Mimir

18cc174

Add integration tests for classic and UTF-8 mode

8f483ca

grobinson-grafana added 5 commits January 25, 2024 11:12

Update CHANGELOG.md

d5b45ec

Add integration tests for matchers/compat metrics

29c8e34

Feedback on CHANGELOG.md

9e4962c

Change utf8-strict-mode to experimental

dd88340

Change log level from Debug to Info

187717b

grobinson-grafana force-pushed the grobinson/utf8-mimir branch from 37f4bb4 to 187717b Compare January 25, 2024 11:13

gotjosh marked this pull request as ready for review January 25, 2024 12:39

gotjosh requested review from a team as code owners January 25, 2024 12:39

gotjosh approved these changes Jan 25, 2024

View reviewed changes

gotjosh merged commit 9c8f825 into main Jan 25, 2024
28 checks passed

gotjosh deleted the grobinson/utf8-mimir branch January 25, 2024 12:41

gotjosh changed the title ~~UTF-8 in Mimir~~ Alertmanager: Support any UTF-8 character in matchers and labels Jan 25, 2024

gotjosh mentioned this pull request Jan 25, 2024

Alertmanager: Support UTF-8 characters in matchers and labels small followup #7221

Merged

grobinson-grafana changed the title ~~Alertmanager: Support any UTF-8 character in matchers and labels~~ Alertmanager: Support UTF-8 Jan 29, 2024

charleskorn mentioned this pull request Jun 4, 2024

Fix changelog entry order #8266

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Alertmanager: Support UTF-8 #6898

Alertmanager: Support UTF-8 #6898

grobinson-grafana commented Dec 12, 2023 •

edited

Loading

CLAassistant commented Dec 12, 2023 •

edited

Loading

gotjosh left a comment

gotjosh left a comment

gotjosh Jan 25, 2024

gotjosh Jan 25, 2024

grobinson-grafana Jan 25, 2024

gotjosh Jan 25, 2024

grobinson-grafana Jan 25, 2024

gotjosh Jan 25, 2024

gotjosh Jan 25, 2024

gotjosh Jan 25, 2024

grobinson-grafana commented Jan 25, 2024

gotjosh left a comment

		@@ -4,6 +4,7 @@

		### Grafana Mimir

		* [CHANGE] Alertmanager: CLI flag `-utf8-strict-mode` has been added, and its default value is set to `true`. #6898

	* [CHANGE] Alertmanager: CLI flag `-utf8-strict-mode` has been added, and its default value is set to `true`. #6898
	* [FEATURE] Alertmanager: Added `-alertmanager.utf8-strict-mode` to control support for any UTF-8 character as part of Alertmanager configuration/API matchers and labels. It's default value is set to `false`. #6898

Alertmanager: Support UTF-8 #6898

Alertmanager: Support UTF-8 #6898

Conversation

grobinson-grafana commented Dec 12, 2023 • edited Loading

What this PR does

Dashboards

Checklist

CLAassistant commented Dec 12, 2023 • edited Loading

gotjosh left a comment

Choose a reason for hiding this comment

gotjosh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

grobinson-grafana commented Jan 25, 2024

gotjosh left a comment

Choose a reason for hiding this comment

grobinson-grafana commented Dec 12, 2023 •

edited

Loading

CLAassistant commented Dec 12, 2023 •

edited

Loading