pytest arg parsing bug

[philippun1]

Type: Bug

Behaviour

Expected vs. Actual

Running a pytest test with pytestArgs results in the error

Error while running tests: test_example
TypeError: Cannot read properties of null (reading 'testsuites')

After removing the pytestArgs from the settings.json, everything runs as expected.

Steps to reproduce:

1. Open a project with a pytest (i.e. https://github.com/philippun1/pytest-vscode-parameters which I created specifically to reproduce this bug)
2. Let VS Code create a settings.json (or use the one from my repo)
3. Add pytestArgs (or use the one from my repo), i.e.

"python.testing.pytestArgs": [
    ".",
    "--test_option",
    "somevalue"
  ],

4. Go to the testing tab and run the test
5. Test Results tab shows the type error
6. Run from the terminal instead gives no error: pytest . --test_option somevalue

Diagnostic data

This test report was made on an Ubuntu machine, but can also be reproduced on a Windows system with the latest VS Code and python extension.

• Python version (& distribution if applicable, e.g. Anaconda): 3.11.2
• Type of virtual environment used (e.g. conda, venv, virtualenv, etc.): Global
• Value of the python.languageServer setting: Default

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

[output is empty]

User Settings

languageServer: "Pylance"

testing
• pytestArgs: "<placeholder>"
• pytestEnabled: true

Extension version: 2023.6.1
VS Code version: Code 1.77.3 (704ed70d4fd1c6bd6342c436f1ede30d1cff4710, 2023-04-12T09:16:52.732Z)
OS version: Linux x64 6.2.0-20-generic
Modes:
Sandboxed: Yes

System Info

Item	Value
CPUs	Intel(R) Core(TM) i5-10210U CPU @ 1.60GHz (8 x 2804)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: disabled_off direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok skia_renderer: enabled_on video_decode: disabled_software video_encode: disabled_software vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: disabled_off
Load (avg)	1, 1, 1
Memory (System)	15.35GB (8.29GB free)
Process Argv	--unity-launch --crash-reporter-id 2d6dab3d-7a47-4211-935d-970af4709a92
Screen Reader	no
VM	0%
DESKTOP_SESSION	ubuntu
XDG_CURRENT_DESKTOP	Unity
XDG_SESSION_DESKTOP	ubuntu
XDG_SESSION_TYPE	wayland

A/B Experiments

vsliv368:30146709
vsreu685:30147344
python383:30185418
vspor879:30202332
vspor708:30202333
vspor363:30204092
vslsvsres303:30308271
vserr242cf:30382550
pythontb:30283811
vsjup518:30340749
pythonptprofiler:30281270
vshan820:30294714
vstes263cf:30335440
vscorecescf:30445987
pythondataviewer:30285071
vscod805:30301674
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
cmake_vspar411:30581797
vsaa593:30376534
pythonvs932:30410667
cppdebug:30492333
vsclangdc:30486549
c4g48928:30535728
dsvsc012:30540252
pynewext54:30695312
azure-dev_surveyone:30548225
3biah626:30602489
pyind779:30671433
f6dab269:30613381
vscrpc:30673769
pythonsymbol12:30671437
6233i204:30730053
vsctsb:30705553
azdwalk:30721579
pythonms35:30701012
pythonfmttextcf:30716743
pythonclientmvcf:30725893
fixshowwlkth:30730052

[eleanorjboyd]

Hi! Yes I am able to replicate what you are seeing. It seems like the extension is trying to run ./.venv/bin/python -m pytest --rootdir . --override-ini junit_family=xunit1 --junit-xml=/var/folders/q8/17wzg0cx7p59c1qfvwxx8yhc0000gn/T/tmp-852821jmb8LyMYWVJ.xml --test_option which does not correctly include the "somevalue" parameter. This is likely due to the parsing we have on our current test design which is buggy. We are in the process of doing a testing rewrite which you can follow along with here. It is almost ready for release and I have already tested it with your use case and it works successfully. Please subscribe to the issue for updates and we will hopefully have more information soon!

[eleanorjboyd]

Hello! We have just finished our testing rewrite and are beginning the roll out to users. If you are able, it would be very helpful to know if your issue still exists on the rewrite! To try it yourself, you need to be on the most recent release of the Python extension and add this setting to your users settings.json "python.experiments.optInto": ["pythonTestAdapter"]. Let me know if the rewrite fixes your issue. Thanks!

[philippun1]

It appears there is no difference if I add the pythonTestAdapter setting or not. Tested both with the latest Python Plugin release (v2023.12.0) and preview release (v2023.13.11881007) with VS Code 1.80.0

Can I somehow verify if the new test adapter is being used?

I updated my test repo with the pythonTestAdapter setting and also some updates to the test itself to make analyzing easier.

[eleanorjboyd]

Hi! You can see if it is enabled for you by setting "python.analysis.logLevel": "Trace", in your user settings then check for 2023-07-05 16:06:27.767 [info] Experiment 'pythonTestAdapter' is active in your python logs. If that message is printed then the rewrite is enabled. I will try and repro this later but if you send over your python logs then I might be able to diagnose it quicker. Thanks for your help!!

[philippun1]

This is my python log if I run the test in the Testing tab:

2023-07-07 22:38:26.077 [warning] Unknown command line option passed into args parser for tests '--test_option'. Please report on https://github.com/Microsoft/vscode-python/issues/new
2023-07-07 22:38:26.077 [info] Running PYTEST execution for the following test ids: test_example.py::test_example
2023-07-07 22:38:26.077 [info] Server listening on port 35477
2023-07-07 22:38:26.084 [info] Running pytests with arguments: /home/phil/.vscode/extensions/ms-python.python-2023.13.11881007/pythonFiles/vscode_pytest/run_pytest_script.py --rootdir /home/phil/Projects/pytest-vscode-parameters --test_option

2023-07-07 22:38:26.084 [info] > /usr/bin/python ~/.vscode/extensions/ms-python.python-2023.13.11881007/pythonFiles/vscode_pytest/run_pytest_script.py --rootdir . --test_option
2023-07-07 22:38:26.084 [info] cwd: .
2023-07-07 22:38:26.199 [info] Test server connected to a client.
2023-07-07 22:38:26.214 [info] Client disconnected

And this if I just run the discovery with the Refresh Tests button:

2023-07-07 22:42:58.794 [info] Discover tests for workspace name: pytest-vscode-parameters - uri: /home/phil/Projects/pytest-vscode-parameters
2023-07-07 22:42:58.794 [info] Running discovery for pytest using the new test adapter.
2023-07-07 22:42:58.808 [info] > /usr/bin/python -m pytest -p vscode_pytest --collect-only . --test_option asd
2023-07-07 22:42:58.808 [info] cwd: .
2023-07-07 22:42:58.912 [info] Test server connected to a client.
2023-07-07 22:42:58.913 [info] Using result resolver for discovery

The log does not change if I set or unset the logLevel to Trace. It feels like the settings are not applied at all?

But we can see that the --test_option asd is added properly on discovery, but not for the real test.

[eleanorjboyd]

hm yes I can see from 2023-07-07 22:42:58.808 [info] > /usr/bin/python -m pytest -p vscode_pytest --collect-only . --test_option asd it is running the new code, maybe it didn't print that it was on the setting since that log is set on startup of the extension. Either way it is running the new stuff. Thanks for checking I will look into this more!!

[philippun1]

There is a workaround for this issue if I use the parameter like this:

At least it is usable now, but it would be great if the arguments would work in the same way as for pytest:
pytest --test_option asd and pytest --test_option=asd both work if I call it directly from a terminal.

[eleanorjboyd]

Hi! Sorry haven't followed up on this in a minute. On further investigation, this is related to a larger issue around settings which the team will be discussing soon. I will update with information after that.

[eleanorjboyd]

I created a proposal with a solution for this problem, would love your input on the above issue! #21845

[melMass]

I'm not 100% sure it's related but the same happens when using pytest-timeout:

Run completed, parsing output
Error while running tests: test_successful_command
TypeError: Cannot read properties of null (reading 'testsuites')

Error while running tests:
TypeError: Cannot read properties of null (reading 'testsuites')

[nsheaps]

Hi all, actually just posted in #22032 about a similar issue and just want to call out that I have args, but when I have -n 8 (part of pytest-xdist for distributing across cpu cores) I can't debug the tests with that error. Just removing the flags works fine and I can debug the tests.

Pytest-xdist explicitly says --pdb is disabled when distributing tests in the docs

[eleanorjboyd]

@nsheaps thank you for your contribution! We will keep this in mind as we work through the proposal!

[schwaerz]

I am getting this error on a very simple pytest project. Cannot get debugging to work.
It does not matter for me whether I use pytestArgs or not. Will always get the same error when trying to debug a test:

Error while running tests: test_git_auth_foo_bar
TypeError: Cannot read properties of null (reading 'testsuites')

Error while running tests:
TypeError: Cannot read properties of null (reading 'testsuites')

Not sure what's going on.

Just tried in another project. Same error there :(

This used to work, as there has been some time while I did not do any python development I cannot say at what time it stopped working.

[eleanorjboyd]

hm interesting- could you open an new bug report since this is likely a different problem? Primarily I would need logs from the python output channel but if you use the report issue command on vscode it will gather all this info for you. Thanks!

[schwaerz]

Ok, will do that, thanks.

[schwaerz]

For reference: #23776