RBAC: access denied on connecting to a notebook - 1.8 pre-release

[NohaIhab]

Bug Description

During the pre-release testing of 1.8, seeing a RBAC: access denied when connecting to a notebook.

To Reproduce

git clone https://github.com/canonical/bundle-kubeflow.git
cd bundle-kubeflow
juju deploy ./releases/latest/edge/bundle.yaml --trust

Environment

microk8s 1.25-strict/stable
juju 3.1/stable
enabled microk8s addons with:
microk8s enable dns hostpath-storage rbac ingress metallb:10.64.140.43-10.64.140.49

Relevant log output

kubectl logs mynb-0 -nadmin
[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
s6-chown: fatal: unable to chown /var/run/s6/etc/cont-init.d/01-copy-tmp-home: Operation not permitted
s6-chown: fatal: unable to chown /var/run/s6/etc/services.d/jupyterlab/run: Operation not permitted
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] 01-copy-tmp-home: executing... 
[cont-init.d] 01-copy-tmp-home: exited 0.
[cont-init.d] done.
[services.d] starting services
[services.d] done.
[I 2023-10-03 07:10:15.093 ServerApp] jupyter_server_mathjax | extension was successfully linked.
[I 2023-10-03 07:10:15.104 ServerApp] jupyterlab | extension was successfully linked.
[I 2023-10-03 07:10:15.105 ServerApp] jupyterlab_git | extension was successfully linked.
[I 2023-10-03 07:10:15.111 ServerApp] nbclassic | extension was successfully linked.
[I 2023-10-03 07:10:15.111 ServerApp] nbdime | extension was successfully linked.
[I 2023-10-03 07:10:15.113 ServerApp] Writing Jupyter server cookie secret to /home/jovyan/.local/share/jupyter/runtime/jupyter_cookie_secret
[I 2023-10-03 07:10:15.351 ServerApp] notebook_shim | extension was successfully linked.
[W 2023-10-03 07:10:15.400 ServerApp] All authentication is disabled.  Anyone who can connect to this server will be able to run code.
[I 2023-10-03 07:10:15.403 ServerApp] notebook_shim | extension was successfully loaded.
[I 2023-10-03 07:10:15.404 ServerApp] jupyter_server_mathjax | extension was successfully loaded.
[I 2023-10-03 07:10:15.406 LabApp] JupyterLab extension loaded from /opt/conda/lib/python3.8/site-packages/jupyterlab
[I 2023-10-03 07:10:15.406 LabApp] JupyterLab application directory is /opt/conda/share/jupyter/lab
[I 2023-10-03 07:10:15.409 ServerApp] jupyterlab | extension was successfully loaded.
[I 2023-10-03 07:10:15.413 ServerApp] jupyterlab_git | extension was successfully loaded.
[I 2023-10-03 07:10:15.418 ServerApp] nbclassic | extension was successfully loaded.
[I 2023-10-03 07:10:15.567 ServerApp] nbdime | extension was successfully loaded.
[I 2023-10-03 07:10:15.568 ServerApp] Serving notebooks from local directory: /home/jovyan
[I 2023-10-03 07:10:15.568 ServerApp] Jupyter Server 1.24.0 is running at:
[I 2023-10-03 07:10:15.568 ServerApp] http://mynb-0:8888/notebook/admin/mynb/lab
[I 2023-10-03 07:10:15.568 ServerApp]  or http://127.0.0.1:8888/notebook/admin/mynb/lab
[I 2023-10-03 07:10:15.568 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).

Additional context

Notebooks version 1.8.0-rc.0

[NohaIhab]

logs from the jupyter-ui workload:

2023-10-03T07:34:26.310Z [jupyter-ui] 2023-10-03 07:34:26,309 | kubeflow.kubeflow.crud_backend.authn | INFO | Handling request for user: admin
2023-10-03T07:34:26.310Z [jupyter-ui] 2023-10-03 07:34:26,310 | kubeflow.kubeflow.crud_backend.csrf | INFO | Skipping CSRF check for safe method: GET
2023-10-03T07:34:26.366Z [jupyter-ui] 2023-10-03 07:34:26,366 | apps.common.utils | INFO | Using config file: /etc/config/spawner_ui_config.yaml
2023-10-03T07:34:26.368Z [jupyter-ui] 10.1.31.161 - - [03/Oct/2023:07:34:26 +0000] "GET /api/namespaces/admin/notebooks HTTP/1.1" 200 1989 "http://10.64.140.43.nip.io/jupyter/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/117.0"
2023-10-03T07:34:34.365Z [jupyter-ui] 2023-10-03 07:34:34,365 | kubeflow.kubeflow.crud_backend.authn | INFO | Handling request for user: admin
2023-10-03T07:34:34.365Z [jupyter-ui] 2023-10-03 07:34:34,365 | kubeflow.kubeflow.crud_backend.csrf | INFO | Skipping CSRF check for safe method: GET
2023-10-03T07:34:34.415Z [jupyter-ui] 2023-10-03 07:34:34,414 | apps.common.utils | INFO | Using config file: /etc/config/spawner_ui_config.yaml
2023-10-03T07:34:34.416Z [jupyter-ui] 10.1.31.161 - - [03/Oct/2023:07:34:34 +0000] "GET /api/namespaces/admin/notebooks HTTP/1.1" 200 1989 "http://10.64.140.43.nip.io/jupyter/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/117.0"
2023-10-03T07:34:35.984Z [jupyter-ui] 2023-10-03 07:34:35,984 | kubeflow.kubeflow.crud_backend.errors.handlers | ERROR | HTTP Exception handled: 401 Unauthorized: No user detected.
2023-10-03T07:34:35.985Z [pebble] Check "up" failure 1823 (threshold 3): received non-20x status code 401

seems like an authorization problem:401 Unauthorized: No user detected

[NohaIhab]

Update: switched backend-mode to development to see debug logs, response from jupyter-ui workload looks ok:

2023-10-03T07:52:37.374Z [jupyter-ui] 10.1.31.161 - - [03/Oct/2023:07:52:37 +0000] "GET /api/namespaces/admin/notebooks HTTP/1.1" 200 3665 "http://10.64.140.43.nip.io/jupyter/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/117.0"
2023-10-03T07:52:42.003Z [jupyter-ui] 2023-10-03 07:52:42,003 | kubeflow.kubeflow.crud_backend.authn | DEBUG | Skipping authentication check in development mode
2023-10-03T07:52:42.004Z [jupyter-ui] 2023-10-03 07:52:42,003 | kubeflow.kubeflow.crud_backend.csrf | INFO | Skipping CSRF check for safe method: GET
2023-10-03T07:52:42.004Z [jupyter-ui] 2023-10-03 07:52:42,004 | kubeflow.kubeflow.crud_backend.authn | DEBUG | User: 'admin' | Headers: 'kubeflow-userid' ''
2023-10-03T07:52:42.004Z [jupyter-ui] 2023-10-03 07:52:42,004 | kubeflow.kubeflow.crud_backend.authz | DEBUG | Skipping authorization check in development mode
2023-10-03T07:52:42.014Z [jupyter-ui] 2023-10-03 07:52:42,014 | kubernetes.client.rest | DEBUG | response body: {"apiVersion":"kubeflow.org/v1beta1","items":[{"apiVersion":"kubeflow.org/v1beta1","kind":"Notebook","metadata":{"annotations":{"notebooks.kubeflow.org/creator":"admin","notebooks.kubeflow.org/last-activity":"2023-10-03T07:10:11Z","notebooks.kubeflow.org/last_activity_check_timestamp":"2023-10-03T07:52:13Z","notebooks.kubeflow.org/server-type":"jupyter"},"creationTimestamp":"2023-10-03T07:10:03Z","generation":2,"labels":{"access-ml-pipeline":"true","app":"mynb"},"managedFields":[{"apiVersion":"kubeflow.org/v1beta1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:notebooks.kubeflow.org/creator":{},"f:notebooks.kubeflow.org/server-type":{}},"f:labels":{".":{},"f:access-ml-pipeline":{},"f:app":{}}},"f:spec":{".":{},"f:template":{".":{},"f:spec":{".":{},"f:serviceAccountName":{},"f:volumes":{}}}}},"manager":"OpenAPI-Generator","operation":"Update","time":"2023-10-03T07:10:03Z"},{"apiVersion":"kubeflow.org/v1beta1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{"f:notebooks.kubeflow.org/last-activity":{},"f:notebooks.kubeflow.org/last_activity_check_timestamp":{}}},"f:spec":{"f:template":{"f:spec":{"f:containers":{}}}}},"manager":"manager","operation":"Update","time":"2023-10-03T07:52:13Z"},{"apiVersion":"kubeflow.org/v1beta1","fieldsType":"FieldsV1","fieldsV1":{"f:status":{".":{},"f:conditions":{},"f:containerState":{".":{},"f:running":{".":{},"f:startedAt":{}}},"f:readyReplicas":{}}},"manager":"manager","operation":"Update","subresource":"status","time":"2023-10-03T07:52:13Z"}],"name":"mynb","namespace":"admin","resourceVersion":"496762","uid":"c568c023-f7aa-48bf-98d4-3644f764bf50"},"spec":{"template":{"spec":{"containers":[{"image":"kubeflownotebookswg/jupyter-scipy:v1.8.0-rc.0","imagePullPolicy":"IfNotPresent","name":"mynb","resources":{"limits":{"cpu":"1200m","memory":"2576980377600m"},"requests":{"cpu":"1","memory":"2Gi"}},"volumeMounts":[{"mountPath":"/dev/shm","name":"dshm"},{"mountPath":"/home/jovyan","name":"mynb-workspace"}]}],"serviceAccountName":"default-editor","volumes":[{"emptyDir":{"medium":"Memory"},"name":"dshm"},{"name":"mynb-workspace","persistentVolumeClaim":{"claimName":"mynb-workspace"}}]}}},"status":{"conditions":[{"lastProbeTime":"2023-10-03T07:52:13Z","lastTransitionTime":"2023-10-03T07:10:13Z","status":"True","type":"Initialized"},{"lastProbeTime":"2023-10-03T07:52:13Z","lastTransitionTime":"2023-10-03T07:10:15Z","status":"True","type":"Ready"},{"lastProbeTime":"2023-10-03T07:52:13Z","lastTransitionTime":"2023-10-03T07:10:15Z","status":"True","type":"ContainersReady"},{"lastProbeTime":"2023-10-03T07:52:13Z","lastTransitionTime":"2023-10-03T07:10:11Z","status":"True","type":"PodScheduled"}],"containerState":{"running":{"startedAt":"2023-10-03T07:10:13Z"}},"readyReplicas":1}}],"kind":"NotebookList","metadata":{"continue":"","resourceVersion":"497008"}}

It's definitely not an authorization issue. UI is still returning 403.

[kimwnasptd]

RBAC: access denied is usually an Istio error, most of the time because there might be a VirtualService blocking traffic (or a misconfiguration on one).

Also, since the issue is in the Notebook Pods then we'll need to focus on the notebook controller

[NohaIhab]

The issue is that in the AuthorizationPolicy created by the Profile controller, the namespace istio-system is hardcoded in:
https://github.com/kubeflow/kubeflow/blob/c6c44923a977d9c5f83d1f641efcbeaeea23cf03/components/profile-controller/controllers/profile_controller.go#L446

This change was introduced in this PR to prevent user impersonation by adding the kubeflow-userid header. While that change fixes a security issue, it introduces an issue where if the istio-ingressgateway service account is in a namespace other than istio-system, then the request will not go through.
To fix the issue, the profile controller needs to be modified with an environment variable where we can pass the namespace, and that env gets injected in the rule.