How to debug the debugger
This page describes how to debug the jupyter extension and debugpy when the user is running a 'debug' command. You should probably read all of the debug topics to familiarize yourself with how debugging works before following the steps below.
Debugging the injected debugger
When the user clicks Debug Cell in a notebook, there are a number of entrypoints (as shown in the sequence diagram on the other wiki page)
You can debug these entry points just like debugging anything in the extension:
- Set a breakpoint
- Launch 'Extension' debugger launch.json entry.
- Perform the user task to start debugging
| Entry Point | Why you might start here | What the cause might be |
|---|---|---|
| Debugging Manager::startDebuggingConfig | Debug cell isn't starting the debugger. This is the main entrypoint and it's what tells VS code to start the debugger | Debug config may not match, so check the config when starting. |
| Kernel Debug Adapter::handleMessage | Debug cell is starting the debugger (or entering debug mode) but breakpoints don't hit or the cell never looks like it's running. | The correct DAP messages are note being sent to the kernel. |
| Kernel Debug Adapter::onIOPubMessage | Debug cell is starting the debugger (or entering debug mode) but breakpoints don't hit or the cell never looks like it's running. | The resulting DAP messages from the debugpy are referencing invalid paths or showing an error. |
| Kernel Debug Adapter::dumpCell | Debug cell is starting the debugger and cell seems to run, but breakpoints don't bind or seem to jump around |
dumpCell may be creating output files not referencable by VS code (like remote paths) or line number translation could be off |
| Kernel Debug Adapter::translate functions | Breakpoints move or don't bind. Wrong files open when stopping | File path translation is generating invalid paths for debugpy or there's a line number translation problem |
Okay so what if all of those entry points look good but debugpy just doesn't seem to respond correctly?
There are three things you can do to debug debugpy:
The easiest thing to do is turn on logging for debugpy with some environment variables:
| Environment Variable | Value |
|---|---|
| PYDEVD_DEBUG | true |
| DEBUGPY_LOG_DIR | directory that already exists |
| PYDEVD_DEBUG_FILE | directory that already exists - use same as previous variable |
Launching VS code with those environment variables will generate files like so in the directory specified:
These files are:
- debugpy.adapter - messages logged by the adapter process (process that is speaking DAP with VS code)
- debugpy.pydevd - messages logged by pydevd (generally has more information)
- debugpy.server - messages logged when debugpy is started in listen mode (more DAP messages)
Opening the adapter log should list out the the DAP messages and their contents. Here's an example:
D+00000.234: Client[1] --> {
"seq": 1,
"type": "request",
"command": "initialize",
"arguments": {
"clientID": "vscode",
"clientName": "Visual Studio Code - Insiders",
"adapterID": "Python Interactive Window Debug Adapter",
"pathFormat": "path",
"linesStartAt1": true,
"columnsStartAt1": true,
"supportsVariableType": true,
"supportsVariablePaging": true,
"supportsRunInTerminalRequest": true,
"locale": "en",
"supportsProgressReporting": true,
"supportsInvalidatedEvent": true,
"supportsMemoryReferences": true
}
}This is:
- a message sent from the client (VS Code) indicated by
Client[1] --> - an initialize message (first DAP message)
A response from the server would look like:
D+00000.281: Server[1] --> {
"pydevd_cmd_id": 502,
"seq": 10,
"type": "response",
"request_seq": 5,
"success": true,
"command": "initialize",
"body": {
"supportsConfigurationDoneRequest": true,
"supportsFunctionBreakpoints": true,
"supportsConditionalBreakpoints": true,
"supportsHitConditionalBreakpoints": true,
"supportsEvaluateForHovers": true,
"exceptionBreakpointFilters": [
{
"filter": "raised",
"label": "Raised Exceptions",
"default": false
},
{
"filter": "uncaught",
"label": "Uncaught Exceptions",
"default": true
},
{
"filter": "userUnhandled",
"label": "User Uncaught Exceptions",
"default": false
}
],
"supportsStepBack": false,
"supportsSetVariable": true,
"supportsRestartFrame": false,
"supportsGotoTargetsRequest": true,
"supportsStepInTargetsRequest": true,
"supportsCompletionsRequest": true,
"completionTriggerCharacters": [],
"supportsModulesRequest": true,
"additionalModuleColumns": [],
"supportedChecksumAlgorithms": [],
"supportsRestartRequest": false,
"supportsExceptionOptions": true,
"supportsValueFormattingOptions": true,
"supportsExceptionInfoRequest": true,
"supportTerminateDebuggee": true,
"supportsDelayedStackTraceLoading": true,
"supportsLoadedSourcesRequest": false,
"supportsLogPoints": true,
"supportsTerminateThreadsRequest": false,
"supportsSetExpression": true,
"supportsTerminateRequest": true,
"supportsDataBreakpoints": false,
"supportsReadMemoryRequest": false,
"supportsDisassembleRequest": false,
"supportsClipboardContext": true,
"supportsDebuggerProperties": true,
"pydevd": {
"processId": 21600
}
}
}You can then verify that the server is responding with the expected messages and that they have expected results. For example you might verify that breakpoints on for the correct files, or that the stack frame response has the correct file in it.
Okay so what if the server isn't sending the correct messages?
You can actually debug debugpy's adapter server. It's the process that is sitting and listening to messages from VS code. This should allow you to see what messages are actually coming from the debuggee (what pydevd is sending below):
You need to set a special environment variable though. This variable lets debugpy debug itself (as of this commit, which will likely be in 1.63 of debugpy).
DEBUGPY_TRACE_DEBUGPY=1
Normally this should be off as it exposes all the hidden details of debugpy to VS code (messes up justmycode).
You need to set this before starting the kernel. There's a number of options to get the kernel to have this environment variable:
- Globally
- Terminal that you start VS code from
-
.envfile that the kernel uses on launche
Once the kernel has that environment variable, you'd start a new instance of VS code and setup a launch.json like so:
{
"name": "Python: Attach using Process Id",
"type": "python",
"request": "attach",
"processId": "${command:pickProcess}",
"justMyCode": false // This has to be false
}the you'd
- Debug a cell in the notebook (in the original VS code instance with the notebook open)
- Wait that one cell to finish debugging (this gets the adapter process started)
- Attach to the
adapterprocess (12428 from before) (in the new VS code) - Set breakpoints (in the new VS code)
- Debug a cell again (in the original VS code)
This should give you something like so:
Debugpy's adapter process is the middleman between VS code and pydevd in the kernel. It handles DAP messages and then forwards them onto pydevd.
In order to set breakpoints, you open the site-packages folder for the kernel you're running. For example, this is where the debugpy installation is for my debugPyLatest conda kernel:
C:\Users\aku91\miniconda3\envs\debugPyLatest\Lib\site-packages\debugpy
Some useful spots to set breakpoints:
| Source location | Site-packages location | What it handles |
|---|---|---|
| Log write | Lib/site-packages/debugpy/common/log.py | Every write to the log goes through here. If you break at this location you can use it to figure out how the adapter is structured. Note, you need to have the other environment variables set before this works. |
| Initialize from Client | Lib/site-packages/debugpy/adapter/clients.py | This is the handler for the initializeRequest from VS code |
| Initialize request to Server | Lib/site-packages/debugpy/adapter/servers.py | Adapter is sending its own initialize request to the server (code injected into the kernel). This is where it asks for 'attach' or 'launch'. |
| Generic event handler | Lib/site-packages/debugpy/adapter/clients.py | Generic event handler. This is the main entry point for events from the server. One such example would be a breakpoint event which should fire when hitting a breakpoint |
What happens if debugging the adapter doesn't help? It's likely the adapter process isn't the source of your problems as it just listens to and sends DAP messages between VS code and the debuggee. The real problem likely lies in the code injected into the debuggee.
Unfortunately there's no way to debug pydevd with a python debugger at the moment, so in order to figure out stuff, you need to add logging.
First set the environment variables for logging, and then open up the site-packages folder for pydevd (it's under where debugpy is)
C:\Users\aku91\miniconda3\envs\debugPyLatest\Lib\site-packages\debugpy\_vendored\pydevd
You can add new logging by following the pattern already in the code. Something like so:
def set_trace_for_frame_and_parents(self, frame, **kwargs):
disable = kwargs.pop('disable', False)
assert not kwargs
while frame is not None:
# Don't change the tracing on debugger-related files
file_type = self.get_file_type(frame)
if file_type is None:
if disable:
pydev_log.debug('Disable tracing of frame: %s - %s', frame.f_code.co_filename, frame.f_code.co_name)
if frame.f_trace is not None and frame.f_trace is not NO_FTRACE:
frame.f_trace = NO_FTRACE
elif frame.f_trace is not self.trace_dispatch:
pydev_log.debug('Set tracing of frame: %s - %s', frame.f_code.co_filename, frame.f_code.co_name)
frame.f_trace = self.trace_dispatch
else:
pydev_log.debug('SKIP set tracing of frame: %s - %s', frame.f_code.co_filename, frame.f_code.co_name)
frame = frame.f_back
del frameThat code is called to turn on or off tracing for different frames in a thread when it stops. You can see it uses pydev_log.debug to trace output. pydev_log should be importable. Just add the debug messages wherever you want and they'll end up in the debugpy.pydevd log