[Pytest] Tests not being discovered due to pytest_report hook

[lubaroli]

Type: Bug

Behaviour

After updating VS Code, VS Code Pytest is no longer discovering tests. Running Pytest on the terminal works as expected and tests are found correctly.

Expected vs. Actual

When running on ssh terminal, within Conda environment:

(kedro_dataiku) xxxxxx@xxxxxx:/ace/src/project_ace$ pytest --disable-warnings --collect-only
================================================= test session starts ==================================================
platform linux -- Python 3.7.16, pytest-3.10.1, py-1.10.0, pluggy-0.13.1
rootdir: /ace/src, inifile:
plugins: dash-1.19.0, anyio-2.2.0, typeguard-2.13.3, mock-1.13.0, cov-2.9.0
collected 1071 items

Output of VS Code after failing to discover tests:

2023-12-13 23:32:55.024 [info] > conda run -n kedro_dataiku --no-capture-output python ~/.vscode-server/extensions/ms-python.python-2023.22.0/pythonFiles/get_output_via_markers.py -m pytest -p vscode_pytest --collect-only
2023-12-13 23:32:55.024 [info] cwd: ./src/project_ace
2023-12-13 23:32:55.907 [info] >>>PYTHON-EXEC-OUTPUT

2023-12-13 23:32:56.172 [info] <<<PYTHON-EXEC-OUTPUT

2023-12-13 23:32:56.172 [error] Traceback (most recent call last):
  File "/home/xxxxx/.vscode-server/extensions/ms-python.python-2023.22.0/pythonFiles/get_output_via_markers.py", line 23, in <module>
    runpy.run_module(moduleName, run_name="__main__", alter_sys=True)
  File "/home/xxxxx/.conda/envs/kedro_dataiku/lib/python3.7/runpy.py", line 205, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/home/xxxxx/.conda/envs/kedro_dataiku/lib/python3.7/runpy.py", line 96, in _run_module_code

2023-12-13 23:32:56.172 [error]     mod_name, mod_spec, pkg_name, script_name)
  File "/home/xxxxx/.conda/envs/kedro_dataiku/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/xxxxx/.conda/envs/kedro_dataiku/lib/python3.7/site-packages/pytest.py", line 91, in <module>
    raise SystemExit(pytest.main())
  File "/home/xxxxx/.conda/envs/kedro_dataiku/lib/python3.7/site-packages/_pytest/config/__init__.py", line 58, in main

2023-12-13 23:32:56.172 [error]     config = _prepareconfig(args, plugins)
  File "/home/xxxxx/.conda/envs/kedro_dataiku/lib/python3.7/site-packages/_pytest/config/__init__.py", line 196, in _prepareconfig

2023-12-13 23:32:56.172 [error]     pluginmanager=pluginmanager, args=args
  File "/home/xxxxx/.conda/envs/kedro_dataiku/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in __call__

2023-12-13 23:32:56.172 [error]     return self._hookexec(self, self.get_hookimpls(), kwargs)

2023-12-13 23:32:56.172 [error]   File "/home/xxxxx/.conda/envs/kedro_dataiku/lib/python3.7/site-packages/pluggy/manager.py", line 93, in _hookexec

2023-12-13 23:32:56.172 [error]     return self._inner_hookexec(hook, methods, kwargs)
  File "/home/xxxxx/.conda/envs/kedro_dataiku/lib/python3.7/site-packages/pluggy/manager.py", line 87, in <lambda>

2023-12-13 23:32:56.172 [error]     firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/home/xxxxx/.conda/envs/kedro_dataiku/lib/python3.7/site-packages/pluggy/callers.py", line 203, in _multicall

2023-12-13 23:32:56.172 [error]     gen.send(outcome)
  File "/home/xxxxx/.conda/envs/kedro_dataiku/lib/python3.7/site-packages/_pytest/helpconfig.py", line 93, in pytest_cmdline_parse

2023-12-13 23:32:56.172 [error]     config = outcome.get_result()
  File "/home/xxxxx/.conda/envs/kedro_dataiku/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result

2023-12-13 23:32:56.172 [error]     raise ex[1].with_traceback(ex[2])
  File "/home/xxxxx/.conda/envs/kedro_dataiku/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall

2023-12-13 23:32:56.172 [error]     res = hook_impl.function(*args)
  File "/home/xxxxx/.conda/envs/kedro_dataiku/lib/python3.7/site-packages/_pytest/config/__init__.py", line 675, in pytest_cmdline_parse

2023-12-13 23:32:56.173 [error]     self.parse(args)
  File "/home/xxxxx/.conda/envs/kedro_dataiku/lib/python3.7/site-packages/_pytest/config/__init__.py", line 845, in parse

2023-12-13 23:32:56.173 [error]     self._preparse(args, addopts=addopts)
  File "/home/xxxxx/.conda/envs/kedro_dataiku/lib/python3.7/site-packages/_pytest/config/__init__.py", line 795, in _preparse

2023-12-13 23:32:56.173 [error]     self.pluginmanager.consider_preparse(args)
  File "/home/xxxxx/.conda/envs/kedro_dataiku/lib/python3.7/site-packages/_pytest/config/__init__.py", line 475, in consider_preparse

2023-12-13 23:32:56.173 [error]     self.consider_pluginarg(opt2)
  File "/home/xxxxx/.conda/envs/kedro_dataiku/lib/python3.7/site-packages/_pytest/config/__init__.py", line 489, in consider_pluginarg

2023-12-13 23:32:56.173 [error]     self.import_plugin(arg)
  File "/home/xxxxx/.conda/envs/kedro_dataiku/lib/python3.7/site-packages/_pytest/config/__init__.py", line 537, in import_plugin

2023-12-13 23:32:56.173 [error]     self.register(mod, modname)
  File "/home/xxxxx/.conda/envs/kedro_dataiku/lib/python3.7/site-packages/_pytest/config/__init__.py", line 306, in register
    ret = super(PytestPluginManager, self).register(plugin, name)
  File "/home/xxxxx/.conda/envs/kedro_dataiku/lib/python3.7/site-packages/pluggy/manager.py", line 126, in register

2023-12-13 23:32:56.173 [error]     self._verify_hook(hook, hookimpl)
  File "/home/xxxxx/.conda/envs/kedro_dataiku/lib/python3.7/site-packages/pluggy/manager.py", line 261, in _verify_hook

2023-12-13 23:32:56.173 [error]     notinspec,
pluggy.manager.PluginValidationError: Plugin 'vscode_pytest' for hook 'pytest_report_teststatus'
hookimpl definition: pytest_report_teststatus(report, config)
Argument(s) {'config'} are declared in the hookimpl but can not be found in the hookspec

Steps to reproduce:

1. Configure tests with Pytest
2. Refresh test discovery

Diagnostic data

• Python version (& distribution if applicable, e.g. Anaconda): 3.7.16
• Type of virtual environment used (e.g. conda, venv, virtualenv, etc.): Conda
• Value of the python.languageServer setting: Pylance

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

XXX

User Settings

condaPath: "<placeholder>"

languageServer: "Pylance"

testing
• cwd: "<placeholder>"
• pytestEnabled: true

Extension version: 2023.22.1
VS Code version: Code 1.85.1 (0ee08df0cf4527e40edc9aa28f4b5bd38bbff2b2, 2023-12-13T09:49:37.021Z)
OS version: Windows_NT x64 10.0.19045
Modes:
Remote OS version: Linux x64 3.10.0-1160.99.1.el7.x86_64
Remote OS version: Linux x64 3.10.0-1160.99.1.el7.x86_64

System Info

Item	Value
CPUs	12th Gen Intel(R) Core(TM) i7-12850HX (24 x 2419)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: enabled_on direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok video_decode: enabled video_encode: enabled vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: enabled
Load (avg)	undefined
Memory (System)	63.67GB (40.83GB free)
Process Argv	--crash-reporter-id d642995a-15db-49b9-871b-7574d0bace4e
Screen Reader	no
VM	0%

Item	Value
Remote	SSH: ace
OS	Linux x64 3.10.0-1160.99.1.el7.x86_64
CPUs	Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz (48 x 2899)
Memory (System)	371.81GB (297.91GB free)
VM	0%

Item	Value
Remote	Container dataiku-base-ace-lbarcelo (reverent_saha) @ ace
OS	Linux x64 3.10.0-1160.99.1.el7.x86_64
CPUs	Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz (48 x 2899)
Memory (System)	371.81GB (297.91GB free)
VM	0%

A/B Experiments

vsliv368:30146709
vsreu685:30147344
python383:30185418
vspor879:30202332
vspor708:30202333
vspor363:30204092
vslsvsres303:30308271
vserr242:30382549
pythontb:30283811
vsjup518:30340749
pythonptprofiler:30281270
vsdfh931:30280409
vshan820:30294714
vstes263:30335439
vscod805cf:30301675
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
vsaa593cf:30376535
pythonvs932:30410667
py29gd2263:30899288
vsclangdf:30486550
c4g48928:30535728
dsvsc012:30540252
azure-dev_surveyone:30548225
282f8724:30602487
89544117:30613380
vscrp:30673768
showlangstatbar:30737416
0bi6i642:30917235
fixshowwlkth:30771522
showindicator:30805244
pythongtdpath:30769146
i26e3531:30792625
welcomedialog:30910333
pythonnosmt12:30797651
pythonidxpt:30866567
pythonnoceb:30805159
asynctok:30898717
dsvsc013:30795093
dsvsc014:30804076
dsvsc015:30845448
pythontestfixt:30902429
pyreplss1:30897532
pythonmypyd1:30879173
pythoncet0:30885854
pythontbext0:30879054
dsvsc016:30899300
dsvsc017:30899301
dsvsc018:30899302
aa_t_chat:30882232

[eleanorjboyd]

Hi! From my understanding what this error message {'config'} are declared in the hookimpl but can not be found in the hookspec is saying is that the config argument is in misalignment. Specifically that the vscode-pytest plugin implementation of pytest_report_teststatus has argument {'config'} but not in the hookspec or the definition outlines what parameters this function should accept.

Referring to pytest documentation here, it shows that config is defined as an argument.

I am seeing from your stdout that your version are Python 3.7.16, pytest-3.10.1, py-1.10.0, pluggy-0.13.1. Pytest is not on version 7.4.3 and Python 3.7 is at end of life so could you try updating both of those because this is likely the problem. Thanks

[vanackere]

Hello,

I'm encountering the same bug (also python 3.7 but cannot upgrade the version since this is for a maintenance branch of a product and that's where dev-containers are useful !)

2023-12-15 09:04:21.566 [info] Experiment 'pythonRecommendTensorboardExtcf' is active
2023-12-15 09:04:21.566 [info] Experiment 'pythonREPLSmartSend' is active
2023-12-15 09:04:21.566 [info] Experiment 'pythonTestAdapter' is active
2023-12-15 09:04:21.566 [info] Default formatter is set to null for workspace /workspaces/wab
2023-12-15 09:04:21.566 [info] Test server listening.
2023-12-15 09:04:21.566 [info] Python interpreter path: /usr/bin/python3
2023-12-15 09:04:32.633 [info] Starting Pylance language server.
2023-12-15 09:04:50.285 [info] Discover tests for workspace name: wab - uri: /workspaces/wab
2023-12-15 09:04:50.285 [info] Running discovery for pytest using the new test adapter.
2023-12-15 09:04:50.286 [info] All environment variables set for pytest discovery for workspace /workspaces/wab: {"PYTHONPATH":"/home/dev/.vscode-server/extensions/ms-python.python-2023.22.1/pythonFiles:srcbuilt:src[...],"VSCODE_AGENT_FOLDER":"/home/dev/.vscode-server","SHELL":"/bin/bash","PWD":"/vscode/vscode-server/bin/linux-x64/0ee08df0cf4527e40edc9aa28f4b5bd38bbff2b2","VSCODE_HANDLES_SIGPIPE":"true","VSCODE_AMD_ENTRYPOINT":"vs/workbench/api/node/extensionHostProcess","VSCODE_HANDLES_UNCAUGHT_ERRORS":"true","VSCODE_NLS_CONFIG":"{\"locale\":\"en\",\"osLocale\":\"en\",\"availableLanguages\":{}}","REMOTE_CONTAINERS_IPC":"/tmp/vscode-remote-containers-ipc-6dc7686f-cb87-4d92-b5fc-e93baef610d2.sock",[...],"REMOTE_CONTAINERS_SOCKETS":"[\"/tmp/vscode-ssh-auth-6dc7686f-cb87-4d92-b5fc-e93baef610d2.sock\",\"/tmp/.X11-unix/X1\"]","TEST_UUID":"2089bcb5-b3e5-433d-a2ba-c27f3960589f","TEST_PORT":"36053"} 

2023-12-15 09:04:50.416 [info] > /usr/bin/python3 -m pytest -p vscode_pytest --collect-only -s --import-mode=append --disable-warnings
2023-12-15 09:04:50.417 [info] cwd: .
2023-12-15 09:04:51.022 [error] Traceback (most recent call last):

2023-12-15 09:04:51.022 [error]   File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main

2023-12-15 09:04:51.022 [error]     
2023-12-15 09:04:51.022 [error] "__main__", mod_spec)
2023-12-15 09:04:51.022 [error] 

2023-12-15 09:04:51.023 [error]   File "/usr/lib/python3.7/runpy.py", line 85, in _run_code

2023-12-15 09:04:51.023 [error]     
2023-12-15 09:04:51.030 [error] exec(code, run_globals)
2023-12-15 09:04:51.030 [error] 

2023-12-15 09:04:51.030 [error]   File "/usr/lib/python3/dist-packages/pytest.py", line 91, in <module>

2023-12-15 09:04:51.031 [error]     
2023-12-15 09:04:51.039 [error] raise SystemExit(pytest.main())

This is probably related to "Experiment 'pythonTestAdapter' is active" ? Anyway it would be nice to have a fix other than "upgrade the python version" if possible ;-)

[vanackere]

I can also confirm that setting "python.experiments.optOutFrom": [ "pythonTestAdapter" ] fixes the issue

[peacefulseeker]

I can also confirm that setting "python.experiments.optOutFrom": [ "pythonTestAdapter" ] fixes the issue

same here(with no obvious error in python output, though). for platform darwin -- Python 3.7.17, pytest-7.4.3, pluggy-1.2.0 environment opting out from pythonTestAdapter helped to discover tests in Vscode Testing section. (make sure to put that in user settings, not workspace settings and reload the VSCode to respect new settings)

Before (and after) that running pytest --collect-only in the terminal had the expected output, so it was the testing UI only that did not pick up discovered tests.

[eleanorjboyd]

@peacefulseeker, are you no longer seeing the same error regarding using python 3.7? If you are not please create a new issue with your project structure and logs so I can better assist, thanks!

[eleanorjboyd]

@vanackere in terms of a fix you can also pin the python extension version which is compatible with 3.7:

Hello! For the python extension we follow python.org and their schedule on which version we support. Here is their supported versions, https://devguide.python.org/versions/, and 3.7 has reached end of life for support. This means new releases of our extension will be compatible back to 3.8 as of now. As a work around, you can pin the version of the python extension which still worked for you with 3.7. You can see in the release notes of v2023.18.0 we dropped Python 3.7 support https://github.com/microsoft/vscode-python/releases so the previous version might be the best version to pin.

[peacefulseeker]

@peacefulseeker, are you no longer seeing the same error regarding using python 3.7? If you are not please create a new issue with your project structure and logs so I can better assist, thanks!

Yes, the issue is gone(for pytest) after I've opted out from pythonTestAdapter experiment, in particular, put the "python.experiments.optOutFrom": [ "pythonTestAdapter" ] in the settings.json

[eleanorjboyd]

Are you able to get it working still in the experiment? We will soon switch to fully using the rewrite as our testing infrastructure which will disable the use of the experiment opt out as a work around.

[vanackere]

@eleanorjboyd : thanks for the hint, we'll try to pin the version in devcontainer.json

My only fear with that approach is that the pinned extension might stop working with future version of VSCode, which will also be a problem for lot of other users working on legacy code base. For exemple Python 3.7 was the default python version on Debian 10 (end-of-life too... but on which one of our products has still a stable supported branch that we must maintain for several more years). How long can we be certain that the pinned extension will still be functional ? If there is a risk that the older extension would stop working, perhaps Microsoft could consider publishing a "Legacy" python extension that would still allow to work with older unsupported versions of python ?

[eleanorjboyd]

Looping in @brettcannon here as he can speak better to the future of our extension and VS Code as it relates to legacy python versions.

[brettcannon]

How long can we be certain that the pinned extension will still be functional ?

No explicit guarantees, but VS Code itself takes API stability seriously, so there's no specific expectation an older version would stop working.

perhaps Microsoft could consider publishing a "Legacy" python extension that would still allow to work with older unsupported versions of python ?

We implicitly already do that via the ability to install old versions of the extension. Otherwise maintaining support w/ older versions of Python becomes too costly time-wise based on the usage we see of old Python versions.

[vanackere]

What is the proper way to pin the version of the extension ?

I've tried putting

"extensions": [
				"[email protected]",

In devcontainer.json but still end up with the latest version.

Also should I also pin ms-python.vscode-pylance to a specific version ?

[peacefulseeker]

Are you able to get it working still in the experiment? We will soon switch to fully using the rewrite as our testing infrastructure which will disable the use of the experiment opt-out as a workaround.

No, I was not able. Makes sense, that at a certain point, the test discovery is expected to be broken
and py3.7 support will be completely dropped, although will still be possible to install previous versions of extensions 👍

[peacefulseeker]

@vanackere , you should be able to use Install another version... in a settings dropdown when you click the cogwheel icon in the extensions search view. Then pick the desired version and reload the editor.

[vanackere]

@peacefulseeker : manual installation works... but that's not really helpful since we need to ensure in the devcontainer.json file that the correct extension version is actually installed and don't want to require that every engineer in our company has to perform a manual setup each time they have to work on the legacy branch of our product...

[brettcannon]

What is the proper way to pin the version of the extension ?

I've tried putting

"extensions": [
				"[email protected]",

In devcontainer.json but still end up with the latest version.

If your version of VS Code is too old then you may be hitting microsoft/vscode#63903 , otherwise it should work and you're hitting a bug in either VS Code or the devcontainer extension.

Also should I also pin ms-python.vscode-pylance to a specific version ?

I honestly don't know for your specific case; Pylance has its own compatibility policy. But since they don't execute Python code to function, I suspect they can be more conservative than us.

[vanackere]

@brettcannon version of vscode is quite recent (Version: 1.85.1 Commit: 0ee08df0cf4527e40edc9aa28f4b5bd38bbff2b2)

This must be a bug in devcontainer extension then, i'll investigate and report

[eleanorjboyd]

@lubaroli, please let me know if your issue has been resolved in this discussion and if we can close the issue. thanks

[lubaroli]

@lubaroli, please let me know if your issue has been resolved in this discussion and if we can close the issue. thanks

Thanks for the quick support @eleanorjboyd, I'm in exactly the same boat as @vanackere, working on legacy systems, but since python 3.7 is reaching EOL, I understand this is not a priority requirement and the issue can be closed. For now I'll pin the devcontainer extension as suggested and hope it is a feasible alternative for as long as it takes us to migrate our system.

[vanackere]

@brettcannon as a follow-up I can confirm that there is a bug that makes extension pinning not work in our case.

I've reduced it to a very small reproducible case and opened an issue at microsoft/vscode#203457, I hope this can be prioritized and fixed soon (hopefully before the testing rewrite lands and fully breaks python 3.7 without any possible workaround for us 😰 )

[eleanorjboyd]

@karthiknadig looping you in on this as you might have more understanding around the extension pinning

[brettcannon]

I've reduced it to a very small reproducible case and opened an issue

Thanks for finding that bug. I can't make any promises around prioritizing that issue, but I will see what I can do.

[eleanorjboyd]

closing as the issue opened on core in regards to extension pinning has been resolved. Thanks!