Update setting up Prometheus docs

[DevSusu]

psschwei: originally opened in serving and moved here. updates needed for docs can be found in this comment: #5217 (comment)

/area monitoring

What version of Knative?

1.7.1 (installed with https://github.com/knative/serving/releases/download/knative-v1.7.1/serving-core.yaml)

Expected Behavior

• install knative-serving
• port-fowarded one of them to debug on prometheus metrics
• pods report metrics described in https://knative.dev/docs/serving/observability/metrics/serving-metrics/

Actual Behavior

kubectl port-forward service/activator-service -n knative-serving 9090:9090
Forwarding from 127.0.0.1:9090 -> 9090
Forwarding from [::1]:9090 -> 9090
Handling connection for 9090
E0914 16:42:51.809345 3007207 portforward.go:400] an error occurred forwarding 9090 -> 9090: error forwarding port 9090 to pod 4f48d8d6708f2f6cc464957b6718d2ec89088951c6430acfb0fb04857e595a94, uid : exit status 1: 2022/09/14 07:42:51 socat[13441] E connect(5, AF=2 127.0.0.1:9090, 16): Connection refused

I had to specify the metrics.backend-destination and metrics.request-metrics-backend-destination to get this working.

Steps to Reproduce the Problem

• follow instructions on https://knative.dev/docs/install/yaml-install/serving/install-serving-with-yaml/#prerequisites
• port-forward one of the components with 9090 port exposed
• curl localhost:9090 (same error with curl localhost:9090/metrics)

apiVersion: v1
kind: ConfigMap
metadata:
  name: config-observability
  namespace: knative-serving
  labels:
    app.kubernetes.io/name: knative-serving
    app.kubernetes.io/component: observability
    app.kubernetes.io/version: "1.7.1"
  annotations:
    knative.dev/example-checksum: "fed4756e"
data: {}

[DevSusu]

related issue: knative/eventing#6509

[DevSusu]

resolved by editing configmap

apiVersion: v1
kind: ConfigMap
metadata:
  name: config-observability
  namespace: knative-serving
  labels:
    app.kubernetes.io/name: knative-serving
    app.kubernetes.io/component: observability
    app.kubernetes.io/version: "1.7.1"
  annotations:
    knative.dev/example-checksum: "fed4756e"
data:
  metrics.backend-destination: prometheus
  metrics.request-metrics-backend-destination: prometheus

[DevSusu]

I think we need to add below block to setting up prometheus part.

https://knative.dev/docs/serving/observability/metrics/collecting-metrics/#setting-up-prometheus

kubectl patch --namespace knative-serving configmap/config-observability \
  --type merge \
  --patch '{"data":{"metrics.backend-destination":"promethuus","metrics.request-metrics-backend-destination":"prometheus"}}'

kubectl patch --namespace knative-serving configmap/config-observability \
  --type merge \
  --patch '{"data":{"metrics.backend-destination":"promethuus","metrics.request-metrics-backend-destination":"prometheus"}}'

[knative-prow]

@DevSusu: The label(s) area/monitoring cannot be applied, because the repository doesn't have them.

In response to this:

/area monitoring

What version of Knative?

1.7.1 (installed with https://github.com/knative/serving/releases/download/knative-v1.7.1/serving-core.yaml)

Expected Behavior

• install knative-serving
• port-fowarded one of them to debug on prometheus metrics
• pods report metrics described in https://knative.dev/docs/serving/observability/metrics/serving-metrics/

Actual Behavior

kubectl port-forward service/activator-service -n knative-serving 9090:9090
Forwarding from 127.0.0.1:9090 -> 9090
Forwarding from [::1]:9090 -> 9090
Handling connection for 9090
E0914 16:42:51.809345 3007207 portforward.go:400] an error occurred forwarding 9090 -> 9090: error forwarding port 9090 to pod 4f48d8d6708f2f6cc464957b6718d2ec89088951c6430acfb0fb04857e595a94, uid : exit status 1: 2022/09/14 07:42:51 socat[13441] E connect(5, AF=2 127.0.0.1:9090, 16): Connection refused

I had to specify the metrics.backend-destination and metrics.request-metrics-backend-destination to get this working.

Steps to Reproduce the Problem

• follow instructions on https://knative.dev/docs/install/yaml-install/serving/install-serving-with-yaml/#prerequisites
• port-forward one of the components with 9090 port exposed
• curl localhost:9090 (same error with curl localhost:9090/metrics)

apiVersion: v1
kind: ConfigMap
metadata:
 name: config-observability
 namespace: knative-serving
 labels:
   app.kubernetes.io/name: knative-serving
   app.kubernetes.io/component: observability
   app.kubernetes.io/version: "1.7.1"
 annotations:
   knative.dev/example-checksum: "fed4756e"
data: {}

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[evankanderson]

/good-first-issue

[knative-prow]

@evankanderson:
This request has been marked as suitable for new contributors.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-good-first-issue command.

In response to this:

/good-first-issue

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[afzal442]

/assign @afzal442

[Rajdeep1311]

Can I help?

[xiangpingjiang]

/assign

[pawarpranav83]

I think we need to add below block to setting up prometheus part.

https://knative.dev/docs/serving/observability/metrics/collecting-metrics/#setting-up-prometheus

kubectl patch --namespace knative-serving configmap/config-observability \
  --type merge \
  --patch '{"data":{"metrics.backend-destination":"promethuus","metrics.request-metrics-backend-destination":"prometheus"}}'

kubectl patch --namespace knative-serving configmap/config-observability \
  --type merge \
  --patch '{"data":{"metrics.backend-destination":"promethuus","metrics.request-metrics-backend-destination":"prometheus"}}'

The config map is to be set before we apply the ServiceMonitors/PodMonitors to collect metrics from Knative, right?

[pawarpranav83]

/assign

[KamalSritha-04]

Please assign me this issue