Test classes no longer work with pytest 6.2.5

[matthew-mcallister]

Behaviour

Expected vs. Actual

This bug occurs with pytest version 6.2.5. All tests should be visible in the Test Explorer, with classes (e.g. class TestWidget) appearing as nested levels in the hierarchy.

However, if a test class is present (or multiple test classes), all other tests are hidden except for the tests in that class.

Steps to reproduce:

1. Define a test file test_example.py with the following text:

   def test_one():
       pass
   
   class TestExample:
       def test_two():
           pass

2. Run test discovery in the Test Explorer.
3. Only test_two is visible.
4. If you delete TestExample, then test_one becomes visible again.

Diagnostic data

• Extension version: v2023.21.13131007
• Python version (& distribution if applicable, e.g. Anaconda): 3.9.12
• Type of virtual environment used (e.g. conda, venv, virtualenv, etc.): Venv
• Value of the python.languageServer setting: Pylance

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

  <Module test_example.py>
    <Function test_outer>
2023-11-09 15:40:25.183 [info] 
    <Class TestClass>
        <Function test_inner>

User Settings

languageServer: "Pylance"

testing
• pytestArgs: "-vv"
• pytestEnabled: true

[TomeSq]

I had the same symptoms.
I reverted back to v2023.8.0 and now I can detect it correctly.

[karthiknadig]

You can opt out of the pytest adapter experiment, if that is causing this issue. You should be able to see which experiment you are in in Output > Python logs.

[karthiknadig]

This seems to be working for me, with the experimental test adapter:

Would it be possible for you to share both logs and settings?

[matthew-mcallister]

Sorry, I figured out the problem. The plugin no longer works with pytest 6.2.5. Updating to the latest version fixed. Updating the issue.

[eleanorjboyd]

Hi @matthew-mcallister, to clarify does it now work with the extension after updating or does this problem still exist? Thanks

[matthew-mcallister]

@eleanorjboyd I meant the issue is fixed by updating to the latest version of pytest, 7.4.3. The issue still exists in the latest version of the vscode extension.

[eleanorjboyd]

the root cause for the issue is the plugin no longer works with pytest 6.2.5 right? If so then that seems like a pytest issue and not an extension issue. Or are you suggesting the issue is that we should have better error reporting?

[matthew-mcallister]

Yes, the issue is that the plugin is incompatible with pytest 6.2.5; this is a breaking change to the plugin. What to do about that is up to the maintainers to decide. Either the extension should continue to work with older versions of pytest to maintain compatibility, or there should be a version check and an error report letting the user know that their version of pytest is not compatible. That way users do not need to grapple with a mysterious bug.

[eleanorjboyd]

@karthiknadig what are you thoughts on the best way to check what version of pytest a user is on and provide an error message as a result?

[eleanorjboyd]

seems like pytest 6.2.5 is still supported and that only versions below 4.6 are no longer supported. This means I should support pytest 6.2.5. Will investigate how to do so

[blubberdiblub]

[...] the best way to check what version of pytest a user is on [...]

FWIW, there are three ways (that I'm aware of) to find out what pytest version is installed. In order of preference, best first, they are:

1. Run the pytest command line tool found on the PATH for executables or, if a specific pytest is configured in the settings, run that. Either way, run it with the --version option and the output will be pytest 7.4.4 or whatever the current version is. Capture that, parse the version and compare against what minimum version you require.

2. If you already have the possibility to run Python code, it's possible to import the pytest module and get the version from that. Execute import pytest and look at pytest.version_tuple or from pytest import version_tuple and look at version_tuple (or any other way you come up with that is equivalent to one of these 2). The value is a tuple containing the major, minor and micro version numbers as integers, so you can easily compare it against a minimally required version.

3. If for some reason you can neither run an executable nor run Python code, there's still the possibility to go through all possible Python import paths and look for a directory that matches pytest-*.dist-info, where the part that matches the asterisk is the version (or alternatively read it from the Version: entry in the pytest-*.dist-info/METADATA file, but the former method should be simpler). The problem with this technique is that apart from a possible PYTHONPATH env variable that may or may not be set, you are unable to reliably determine what the set of possible Python import paths is without executing Python and would have to resort to guessing / using a predefined list. Therefore avoid this technique like the plague and go for one of the first two unless you absolutely have no other choice.

[eleanorjboyd]

Hello! Just as an update here I discussed with the pytest developers: pytest-dev/pytest#12253 and agree with their assessment that pytest 7.0.0 should be the lowest version we will support in the extension at this time. I will update the docs to reflect that, thanks