From d389802464d170b6cbac26f17c8d5e31d3d8bc16 Mon Sep 17 00:00:00 2001 From: David Brochart Date: Mon, 9 Jan 2023 18:31:21 +0100 Subject: [PATCH] Review --- kernel-subshells/kernel-subshells.md | 65 +++++++++++++++++++++------- 1 file changed, 49 insertions(+), 16 deletions(-) diff --git a/kernel-subshells/kernel-subshells.md b/kernel-subshells/kernel-subshells.md index 654ab802..aadc1ee1 100644 --- a/kernel-subshells/kernel-subshells.md +++ b/kernel-subshells/kernel-subshells.md @@ -8,19 +8,25 @@ date-started: 2022-12-15 # Summary -This JEP introduces kernel sub-shells to allow for concurrent code execution. This is made possible -by defining new control channel messages, as well as a new shell ID field in shell channel messages. +This JEP introduces kernel sub-shells to allow for concurrent shell requests. This is made possible +by defining new control channel messages, as well as a new shell ID field in shell messages. # Motivation Users have been asking for ways to interact with a kernel while it is busy executing CPU-bound code, for the following reasons: -- inspect the kernel's state to check the progress or debug a long-running computation. -- visualize some intermediary result before the final result is computed. +- inspect the kernel's state to check the progress or debug a long-running computation (e.g. + through a variable explorer). +- visualize intermediary results before the final result is computed. +- request [completion](https://jupyter-client.readthedocs.io/en/stable/messaging.html#completion) or + [introspection](https://jupyter-client.readthedocs.io/en/stable/messaging.html#introspection). +- process + [Comm messages](https://jupyter-client.readthedocs.io/en/stable/messaging.html#custom-messages) + immediately (e.g. for widgets). Unfortunately, it is currently not possible to do so because the kernel cannot process other -[execution requests](https://jupyter-client.readthedocs.io/en/stable/messaging.html#execute) until -it is idle. The goal of this JEP is to offer a way to run code concurrently. +[shell requests](https://jupyter-client.readthedocs.io/en/stable/messaging.html#messages-on-the-shell-router-dealer-channel) +until it is idle. The goal of this JEP is to offer a way to process shell requests concurrently. # Proposed Enhancement @@ -38,8 +44,9 @@ for: - deleting a sub-shell, - listing existing sub-shells. -A sub-shell should be advertised to the client with a shell ID, which must be sent along with -further messages on the shell channel in order to target a sub-shell. This allows any other client +A sub-shell should be identified with a shell ID, either provided by the client in the sub-shell +creation request, or given by the kernel in the sub-shell creation reply. The shell ID of the +targeted sub-shell must then be sent along with any shell message. This allows any other client (console, notebook, etc.) to use this sub-shell. If no shell ID is sent, the message targets the main shell. Sub-shells are thus multiplexed on the shell channel through the shell ID, and it is the responsibility of the kernel to route the messages to the target sub-shell according to the shell @@ -55,7 +62,14 @@ is the responsibility of users to not corrupt the kernel state with non thread-s ## Create sub-shell -Message type: `create_subshell_request`: no content. +Message type: `create_subshell_request`: + +```py +content = { + # Optional, the ID of the sub-shell if specified by the client. + 'shell_id': str +} +``` Message type: `create_subshell_reply`: @@ -64,7 +78,8 @@ content = { # 'ok' if the request succeeded or 'error', with error information as in all other replies. 'status': 'ok', - # The ID of the sub-shell. + # The ID of the sub-shell, same as in the request if specified by the client, given by the + # kernel otherwise. 'shell_id': str } ``` @@ -102,10 +117,28 @@ content = { } ``` -# Points of discussion +# Behavior + +## Kernels not supporting sub-shells + +The following requests should be ignored: `create_subshell_request`, `delete_subshell_request` and +`list_subshell_request`. A `shell_id` passed in any shell message should be ignored. This ensures +that existing kernels don't need any change to be compatible with the kernel protocol changes +required by this JEP. + +This means that all shell messages are processed in the main shell, i.e. sequentially. + +Since sub-shells are basically a "no-op", the behavior around +[kernel restart](https://jupyter-client.readthedocs.io/en/stable/messaging.html#kernel-shutdown) and +[kernel interrupt](https://jupyter-client.readthedocs.io/en/stable/messaging.html#kernel-interrupt) +is unchanged. + +## Kernels supporting sub-shells + +A sub-shell request may be processed concurrently with other shells. Within a sub-shell, requests +are processed sequentially. -The question of sub-shell ownership and life cycle is open, in particular: -- Is a sub-shell responsible for deleting itself, or can a shell delete other sub-shells? -- Can a sub-shell create other sub-shells? -- Does sub-shells have the same rights as the main shell? For instance, should they be allowed to - shut down or restart the kernel? +A [kernel restart](https://jupyter-client.readthedocs.io/en/stable/messaging.html#kernel-shutdown) +should delete all sub-shells. A +[kernel interrupt](https://jupyter-client.readthedocs.io/en/stable/messaging.html#kernel-interrupt) +should interrupt the main shell and all sub-shells.