jupyter ui stalls and becomes unusable

[nishant-dash]

Bug Description

Jupyter ui just starts stalling and I can even stop a notebook. I get either internal errors in the ui or

0: Http failure response for https://<REDACTED>/jupyter/api/namespaces/admin/notebooks/test-dash-2: 0 Unknown Error

After recreating the jupyter ui pod, things work.

To Reproduce

N/A

Environment

intermediate version of kf 1.7
intermediate version of jupyter ui

Relevant Log Output

In the jupyter ui container, I see tracebacks of 

2024-02-21T15:44:27.541Z [jupyter-ui] 2024-02-21 15:44:27,541 | apps.common.utils | INFO | Using config file: /etc/config/spawner_ui_config.yaml
2024-02-21T15:44:27.563Z [jupyter-ui] 2024-02-21 15:44:27,563 | apps.common.utils | INFO | Using config file: /etc/config/spawner_ui_config.yaml
2024-02-21T15:44:27.586Z [jupyter-ui] 2024-02-21 15:44:27,586 | apps.common.utils | INFO | Using config file: /etc/config/spawner_ui_config.yaml
2024-02-21T15:44:27.609Z [jupyter-ui] 2024-02-21 15:44:27,609 | apps.common.utils | INFO | Using config file: /etc/config/spawner_ui_config.yaml
2024-02-21T15:44:27.609Z [jupyter-ui] 2024-02-21 15:44:27,609 | kubeflow.kubeflow.crud_backend.errors.handlers | ERROR | Caught and unhandled Exception!
2024-02-21T15:44:27.610Z [jupyter-ui] 2024-02-21 15:44:27,609 | kubeflow.kubeflow.crud_backend.errors.handlers | ERROR | 'image'
2024-02-21T15:44:27.610Z [jupyter-ui] Traceback (most recent call last):
2024-02-21T15:44:27.610Z [jupyter-ui]   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1820, in full_dispatch_request
2024-02-21T15:44:27.610Z [jupyter-ui]     rv = self.dispatch_request()
2024-02-21T15:44:27.610Z [jupyter-ui]   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1796, in dispatch_request
2024-02-21T15:44:27.610Z [jupyter-ui]     return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
2024-02-21T15:44:27.610Z [jupyter-ui]   File "/src/apps/common/routes/get.py", line 47, in get_notebooks
2024-02-21T15:44:27.610Z [jupyter-ui]     contents = [utils.notebook_dict_from_k8s_obj(nb) for nb in notebooks]
2024-02-21T15:44:27.610Z [jupyter-ui]   File "/src/apps/common/routes/get.py", line 47, in <listcomp>
2024-02-21T15:44:27.610Z [jupyter-ui]     contents = [utils.notebook_dict_from_k8s_obj(nb) for nb in notebooks]
2024-02-21T15:44:27.610Z [jupyter-ui]   File "/src/apps/common/utils.py", line 135, in notebook_dict_from_k8s_obj
2024-02-21T15:44:27.610Z [jupyter-ui]     "image": cntr["image"],
2024-02-21T15:44:27.610Z [jupyter-ui] KeyError: 'image'

Additional Context

Thanks @pedrofragola for pointing to kubeflow/kubeflow#6886

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5362.

This message was autogenerated

[nishant-dash]

maybe the python packages are doing something different now? and this spawner ui config yaml is incompatible?
(given that this is an older version of jupyter ui)

[pedrofragola]

plz check the issue kubeflow/kubeflow#6886

[nishant-dash]

Hi given what @pedrofragola linked, this seems like a serious issue since any user can easily or inadvertently DOS the entire jupyter ui service