Observe and Prometheus/Alertmanager API queries broken after upgrade 4.12 -> 4.16

[cannibalisticcow]

Describe the bug
A user that has access to view the Observe panel and make queries to the endpoints;
/api/prometheus/api/v1/rules
and
/api/alertmanager-user-workload/api/v2/alerts

On version 4.12 can no longer do this when the cluster is upgraded to 4.15.
The Observe panel shows Forbidden and when looking at the network response the problem seems to be;
Forbidden (user=robot, verb=get, resource=prometheuses, subresource=api)
When querying the user-workload endpoint the problem is
Forbidden (user=robot, verb=get, resource=alertmanagers, subresource=api)

I've been able to work around the problem by creating a new cluserrole and clusterrolebinding but feel like as this worked previously it should continue to work post upgrade. The user has a number of default custom roles applied to allow cluster metrics and state to be viewed.
Default Roles applied

rules:
  - verbs:
      - get
    apiGroups:
      - ''
    resources:
      - namespaces
  - verbs:
      - get
      - list
      - watch
    apiGroups:
      - config.openshift.io
    resources:
      - clusterversions
      - infrastructures
  - verbs:
      - watch
      - list
    apiGroups:
      - ''
    resources:
      - events

rules:
  - verbs:
      - get
      - list
      - watch
    apiGroups:
      - ''
    resources:
      - nodes
  - verbs:
      - list
      - get
      - watch
    apiGroups:
      - metrics.k8s.io
    resources:
      - nodes
      - pods
  - verbs:
      - get
      - list
      - watch
    apiGroups:
      - ''
    resources:
      - namespaces
  - verbs:
      - list
    apiGroups:
      - ''
    resources:
      - pods
      - services
  - verbs:
      - list
    apiGroups:
      - monitoring.coreos.com
    resources:
      - servicemonitors
      - podmonitors
  - verbs:
      - get
      - list
      - watch
    apiGroups:
      - config.openshift.io
    resources:
      - clusterversions
      - clusteroperators
      - infrastructures
  - verbs:
      - watch
    apiGroups:
      - events.k8s.io
    resources:
      - events
  - verbs:
      - list
    apiGroups:
      - machineconfiguration.openshift.io
      - machine.openshift.io
      - autoscaling.openshift.io
    resources:
      - machineconfigpools
      - machines
      - machinesets
      - machineautoscalers
      - machinehealthchecks
      - machineconfigs
  - verbs:
      - watch
      - list
    apiGroups:
      - ''
    resources:
      - events
      - persistentvolumeclaims
  - verbs:
      - watch
      - list
    apiGroups:
      - network.openshift.io
    resources:
      - netnamespaces
  - verbs:
      - watch
      - list
    apiGroups:
      - operators.coreos.com
    resources:
      - clusterserviceversions

Work around

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: monitoring-api-fix
rules:
  - verbs:
      - get
    apiGroups:
      - monitoring.coreos.com
    resources:
      - prometheuses/api
      - alertmanagers/api
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: monitoring-api-fix
subjects:
  - kind: Group
    apiGroup: rbac.authorization.k8s.io
    name: 'system:authenticated'
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: monitoring-api-fix

Version
4.15.0-0.okd-2024-03-10-010116

How reproducible
We have multiple OKD clusters, all of the ones that have been upgraded from 4.12 to 4.15 are displaying this problem. The ones that are still at 4.12 continue to work.
I've not had the opportunity yet to see if this is a problem that was introduced in .13, .14, or .15

[cannibalisticcow]

Observe page functions normally in 4.13.0-0.okd-2023-09-30-084937

[cannibalisticcow]

Tested on 4.14.0-0.okd-2024-01-06-084517 Other than a bug with the plug-in rbac is working correctly for accessing the Observe page and querying the above endpoints.

[cannibalisticcow]

Breaks from 4.15.0-0.okd-2024-02-10-035534

[cannibalisticcow]

A 4.16 cluster has now been built too and the problem also exists there.
4.16.0-0.okd-scos-2024-08-14-110444

[JaimeMagiera]

Thanks for testing. We'll take a look.

[Greenwavemonster]

hi guys
We just had the same issue updating our OpenShift cluster to 4.16. We think it could have something to to with the change from OAuth to kube-rbac-proxy for monitoring component.

In our case Grafana failed whith the exact same error message as @cannibalisticcow mentioned. Grafana used a custom SA and custom clusterrole with get,list,watch on all namespaces. This is because the cluster-monitoring-view clusterrole does not have list and watch.
After the update however we noticed that the cluster-monitoring-view has this new part:

- apiGroups:
  - monitoring.coreos.com
  resourceNames:
  - k8s
  resources:
  - prometheuses/api
  verbs:
  - get
  - create
  - update

which is not on older cluster.
So we just created a additional binding for the SA to link to the above meintined view role and the problem was solved.

Hope this helps :)