Explore how we can make test discovery more automatic

[luabud]

Based on microsoft/vscode#210152

Right now we require manual configuration for test discovery to be performed, but are there ways we can make it more automatic?

I believe there are currently a few challenges:
1- we don't know which framework the tests should be configured with, and there may be a lot of different configs and arguments that people need to pass so tests can successfully be discovered. If we tried to something automatically without requiring user config, most likely we would show a failure on the test view.
2- pytest needs to be installed in user's environment for it to perform discovery. When people configure their tests to use pytest and it's not installed, we automatically install it on their behalf. We ofc can't do that automatically on load though, so we'd need to figure out a way to handle that part.
3- Some users who are running tests in the terminal are not using the same environment as the one we select for the workspace, which can also cause some confusion if we tried to do discovery automatically, as we'd likely fail due to the wrong environment being selected, even though running tests in the terminal succeeds. It's a problem that already exists, but since we don't open the test view by default nor try to perform discovery automatically, it's less noticeable

In any case, this is likely currently blocked by #21845 and potentially #12075?

[isidorn]

Thanks for creating this issue.

1. Could the test view show some "welcome experience" - so the user can choose the framework. Or could we detect the framework automatically based on workspace structure?
2. Could we do this from the test view? So not fully automatic, but to offer installing pytest to the user. Either via some contributed button (or if not possible, via a notification)

For #12075 is something needed from the VS Code side? Or you have all the testing api you need?

fyi @connor4312

[connor4312]

One idea I had was to show an editor button (like the one we show in the launch.json, below) if the file matches the common test_*.py, *_test.py, etc. patterns and the user has not configured tests in their environment or dismissed the button.

Could the test view show some "welcome experience" - so the user can choose the framework. Or could we detect the framework automatically based on workspace structure?

Python does offer a welcome experience in that view when it's installed

[eleanorjboyd]

One idea I had was to show an editor button (like the one we show in the launch.json, below) if the file matches the common test_*.py, *_test.py, etc. patterns and the user has not configured tests in their environment or dismissed the button.

Could the test view show some "welcome experience" - so the user can choose the framework. Or could we detect the framework automatically based on workspace structure?

Python does offer a welcome experience in that view when it's installed

I like this idea of the pop-up on a test file- would it be possible to introduce something so the user could dismiss?

[connor4312]

Currently the editor button only supports a single action. But you could have a 'configure later' option in the quickpick, or PR in support for multiple actions there https://github.com/microsoft/vscode/blob/cf7f91d9bbdb4787de5b2828e1ec956d05790926/src/vs/workbench/browser/codeeditor.ts#L171