Inaccessible tracing URL is exposed over cos_agent if no tracing backend is connected

[mmkay]

Bug Description

When a charm is related to grafana-agent without a connected tracing backend and requests a tracing endpoint then cos-agent exposes an inaccessible tracing URL.

https://github.com/canonical/grafana-agent-operator/blob/main/lib/charms/grafana_agent/v0/cos_agent.py#L990 exposes all the requested receivers, regardless if they were enabled in grafana agent or not.

To Reproduce

Deploy the following bundle in a machine model (don't relate it to tempo / COS):

default-base: [email protected]/stable
applications:
  grafana-agent:
    charm: grafana-agent
    channel: latest/edge
    revision: 332
  postgresql:
    charm: postgresql
    channel: 14/edge
    revision: 530
    num_units: 1
    to:
    - "2"
    constraints: arch=amd64
    storage:
      pgdata: rootfs,1,1024M
machines:
  "2":
    constraints: arch=amd64
relations:
- - postgresql:cos-agent
  - grafana-agent:cos-agent

Observe logs to see an error described in log output. cat /etc/grafana-agent.yaml also shows that the tracing section is empty, so grafana-agent doesn't expose any tracing endpoints:

(...)
          juju_unit: postgresql/1
        targets:
        - localhost:9187
  global:
    scrape_interval: 1m
    scrape_timeout: 10s
  wal_directory: /tmp/agent/data
server:
  log_level: info
traces: {}

Environment

Tested on LXD on multipass (charm-dev blueprint), Juju 3.5.5

Relevant log output

Log in postgresql charm, pointing it received a tracing URL: 


unit-postgresql-2: 21:55:54 ERROR unit.postgresql/2.juju-log database-peers:6: unexpected error while flushing span batch from buffer
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-postgresql-2/charm/venv/urllib3/connection.py", line 199, in _new_conn
    sock = connection.create_connection(
  File "/var/lib/juju/agents/unit-postgresql-2/charm/venv/urllib3/util/connection.py", line 85, in create_connection
    raise err
  File "/var/lib/juju/agents/unit-postgresql-2/charm/venv/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-postgresql-2/charm/venv/urllib3/connectionpool.py", line 789, in urlopen
    response = self._make_request(
  File "/var/lib/juju/agents/unit-postgresql-2/charm/venv/urllib3/connectionpool.py", line 495, in _make_request
    conn.request(
  File "/var/lib/juju/agents/unit-postgresql-2/charm/venv/urllib3/connection.py", line 441, in request
    self.endheaders()
  File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1038, in _send_output
    self.send(msg)
  File "/usr/lib/python3.10/http/client.py", line 976, in send
    self.connect()
  File "/var/lib/juju/agents/unit-postgresql-2/charm/venv/urllib3/connection.py", line 279, in connect
    self.sock = self._new_conn()
  File "/var/lib/juju/agents/unit-postgresql-2/charm/venv/urllib3/connection.py", line 214, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f79b10c6230>: Failed to establish a new connection: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-postgresql-2/charm/venv/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
  File "/var/lib/juju/agents/unit-postgresql-2/charm/venv/urllib3/connectionpool.py", line 843, in urlopen
    retries = retries.increment(
  File "/var/lib/juju/agents/unit-postgresql-2/charm/venv/urllib3/util/retry.py", line 519, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='juju-7e29af-2.lxd', port=4318): Max retries exceeded with url: /v1/traces (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f79b10c6230>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-postgresql-2/charm/lib/charms/tempo_coordinator_k8s/v0/charm_tracing.py", line 483, in flush
    out = self.exporter._export(span)  # type: ignore
  File "/var/lib/juju/agents/unit-postgresql-2/charm/venv/opentelemetry/exporter/otlp/proto/http/trace_exporter/__init__.py", line 114, in _export
    return self._session.post(
  File "/var/lib/juju/agents/unit-postgresql-2/charm/venv/requests/sessions.py", line 637, in post
    return self.request("POST", url, data=data, json=json, **kwargs)
  File "/var/lib/juju/agents/unit-postgresql-2/charm/venv/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/var/lib/juju/agents/unit-postgresql-2/charm/venv/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/var/lib/juju/agents/unit-postgresql-2/charm/venv/requests/adapters.py", line 700, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='juju-7e29af-2.lxd', port=4318): Max retries exceeded with url: /v1/traces (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f79b10c6230>: Failed to establish a new connection: [Errno 111] Connection refused'))
unit-postgresql-2: 21:55:54 ERROR unit.postgresql/2.juju-log database-peers:6: failed flushing spans; buffer preserved
unit-postgresql-2: 21:55:54 ERROR unit.postgresql/2.juju-log database-peers:6: buffer flush FAILED

### Additional context

_No response_