PodMonitors for Ping Source Controller result in TargetDown alerts from Prometheus

[chrismellard]

Describe the bug

Deploying Knative Eventing and Prometheus Operator with the PodMonitor examples from the sandbox repo results in TargetDown alerts when the metrics are unscrapeable by Prometheus. The pingsource-mt-adapter Deployment supposedly exposes metrics on port 9090 yet when I attempt to port-forward to this port the tunnel crashes when I attempt to retrieve anything over this port.

Expected behavior
I expect that the pingsource-mt-adapter Pods that define a metrics port would expose these when a http request is issued against this port

To Reproduce
Vanilla knative eventing install + Prometheus Operator + PodMonitor as per your docs

Knative release version
1.6.0

[DevSusu]

Same here with knative-serving 1.7.1 knative/docs#5217

[DevSusu]

@chrismellard I've got metrics working by editing config-observability as below.

https://github.com/knative/serving/blob/main/config/core/configmaps/observability.yaml#L94

    # metrics.backend-destination field specifies the system metrics destination.
    # It supports either prometheus (the default) or opencensus.
    metrics.backend-destination: prometheus
    # metrics.request-metrics-backend-destination specifies the request metrics
    # destination. It enables queue proxy to send request metrics.
    # Currently supported values: prometheus (the default), opencensus.
    metrics.request-metrics-backend-destination: prometheus

apiVersion: v1
kind: ConfigMap
metadata:
  name: config-observability
  namespace: knative-serving
  labels:
    app.kubernetes.io/name: knative-serving
    app.kubernetes.io/component: observability
    app.kubernetes.io/version: "1.7.1"
  annotations:
    knative.dev/example-checksum: "fed4756e"
data:
  metrics.backend-destination: prometheus
  metrics.request-metrics-backend-destination: prometheus

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

[pierDipi]

Fixed by #6615