Don't expose tracing endpoints over cos_agent when backend isn't available #228
+93
−66
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue
When grafana-agent isn't related to a tracing backend, but a charm relates to grafana-agent over
cos_agent
requesting tracing receivers, charm event hooks take a significant amount of time as the charm tries to send traces to an endpoint that isn't open in grafana-agent.Solution
Return
null
when tracing isn't available with a reasonable log message. Fixes #219.Context
Backward compatibility was tested using https://github.com/canonical/postgresql-operator, observed behaviour:
cos_agent
connects to new grafana-agent and requests tracing endpoints when grafana-agent isn't connected to a tracing backend, its logs will show that the databag wasn't validated. As tracing receivers are the only part grafana-agent returns to the charm in this databag, it doesn't cause any issues, but the charm doesn't try to send charm traces, as desired.cos_agent
connects to new grafana-agent, the following log line will appear:There's also an open question of whether grafana-agent (and grafana-agent-k8s) should be put into blocked state if a charm requests tracing, but grafana-agent isn't connected to the tracing backend. So far I think this should be a separate issue (and will create one once we agree on it), but I might be convinced otherwise.
Testing Instructions
Deploy the following bundle with a local build of grafana-agent from this branch.
Observe that connection exceptions in postgresql no longer appear in
juju debug-log
.Upgrade Notes