Custom Test Discovery Option

[juandiegopalomino]

Type: Feature Request

Behaviour

Expected vs. Actual

Hi there!

So right now, the python test suite discovery relies on pytest running with the --collect-only flag. For normal projects, this works fine as this pytest command would take 1-2 seconds. But if you happen to have a really big python project with 80K tests and counting like in my company, Rippling, then the test collection takes over 10 minutes, and usually breaks the editor with too much data.

Here for the typescript version

And here for the python version

We really need to overcome this if we are to officially adopt VS Code as our editor for the company.

Before making a whole private extension to support this, I would like to double check one more time (I first checked in October, but I know things might have changed since then) if there is anything official coming up for this. This problem is really like the last big blocker, and I'm sure that other companies with large codebases are facing it right now too.

Steps to reproduce:

1. Get a codebase with ~80K tests and ~300 dependencies.
2. Setup the venv everything usual there.
3. Do the regular flow to configure test runner with pytest.
4. Watch as your editor stalls for 10 minutes and then enters a broken state.

Diagnostic data

• Python version (& distribution if applicable, e.g. Anaconda): 3.10.11
• Type of virtual environment used (e.g. conda, venv, virtualenv, etc.): Poetry
• Value of the python.languageServer setting: Default

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

XXX

User Settings

languageServer: "Pylance"

testing
• pytestArgs: "<placeholder>"
• pytestEnabled: true

experiments
• optInto: ["pythonTestAdapter"]

Extension version: 2024.2.1
VS Code version: Code 1.87.0 (Universal) (019f4d1419fbc8219a181fab7892ebccf7ee29a2, 2024-02-27T23:42:56.944Z)
OS version: Darwin arm64 23.3.0
Modes:

[drothlis]

In #21922 (comment) @brettcannon wrote:

You make a pytest plug-in and if you can get pytest to do what you want then that will be doable as we are working on letting you specify the command we pass to pytest for discovery separate from execution. But I don't know if pytest specifically gives you that level of flexibility to short-circuit discovery and take it over completely.

Yes, it does. We wrote a pytest plug-in that does custom collection and execution:
https://david.rothlis.net/running-tests-on-remote-hardware-from-your-ide/

(I'm replying here because #21922 is locked to new comments.)

[drothlis]

@juandiegopalomino but do be aware that VSCode's Python plugin occasionally breaks our use of pytest, e.g. #8242/#8425 (our workaround: stb-tester/stbt-rig@89f7a27) and #22076/#22885/#22999 (our workaround: stb-tester/stbt-rig@a88df29). I suppose twice in 5 years isn't too bad, and in both cases we were able to implement a work-around, but it did break the VSCode integration for our users.

The vscode-python team also don't seem to test against older (but still supported) versions of pytest? e.g. #22457. But this one didn't affect us.

I'm just saying you might want to consider writing a custom VSCode plugin; maybe VSCode's plugin API is more stable (but I have no experience with that).

[juandiegopalomino]

@drothlis hi!

Wow! Your code is awesome, it helps so much! One question though-- it looks like the pytest plugin will still execute for all the python files in the repo (i.e. it will go one by one through all of my files). That might still be too long, as that would still be thousands of files 😅 . Is there anyway to say "just run this on the currently opened files?"?

[drothlis]

One question though-- it looks like the pytest plugin will still execute for all the python files in the repo (i.e. it will go one by one through all of my files). That might still be too long, as that would still be thousands of files 😅 .

It's just grepping for ^def\s+(test_[a-zA-Z0-9_]*) so it should be fast, even for thousands of files. I suppose it could end up reporting more tests than VSCode can handle?

Is there anyway to say "just run this on the currently opened files?"?

You could try adding "${file}" to the "python.testing.pytestArgs" in .vscode/settings.json (see Predefined variables) but I don't know if that's supported in pytestArgs (see Is variable substitution supported in User and Workspace settings?).

[brettcannon]

The vscode-python team also don't seem to test against older (but still supported) versions of pytest?

We do now:

vscode-python/.github/workflows/pr-check.yml

Line 98 in 209d6bd

pytest-version: ['pytest', 'pytest@pre-release', 'pytest==6.2.0']

[eleanorjboyd]

hi @juandiegopalomino, looks like you got some awesome suggestions from @drothlis, thanks @drothlis for helping out! Two points:

1. I am surprised your editor freezes, can you explain more about the steps you get to cause this and what you mean by "freezes". I think given this case an easy fix would be to make the editor usable as discovery happens so your flow isn't interrupted.
2. I have begun looking into a setting where refresh for test discovery when you save a file would only run discovery on this particular file. Say we added a setting so discovery didn't happen when you clicked on the testing icon + this per file discovery; would this help?

[vltr]

@eleanorjboyd I'm going to pry a bit in here, but option two seems very interesting! 😅

[eleanorjboyd]

Hi! If you are interested definitely give the original issue an upvote as we decide prioritization of tasks based on community interest (as measured by upvote count) #4586

[github-actions]

Because we have not heard back with the information we requested, we are closing this issue for now. If you are able to provide the info later on, then we will be happy to re-open this issue to pick up where we left off.

Happy Coding!