Add separate python.testing.pytestArgs settings for regular, debug and discovery runs

[mpekurin]

Evolved from #21190.

Related to #693, #19809, #19985 and some others.

Some of the pytest plugins may change how it works (e.g. running in parallel via pytest-xdist) or do some actions that are required on regular test runs but are unwanted on debug or discovery runs (e.g. generating a coverage report via pytest-cov). Currently vscode provides a single setting for configuring the args, and they are used for all the kinds of runs. Sometimes such behavior is inconvenient. Separating the single setting into 3 (for regular runs, for debug runs and for discovery runs) would make configuring pytest much more flexible.

Something like

"python.testing.pytestArgs": {
    "run": [], // equal to the current "python.testing.pytestArgs": []
    "debug": [],
    "discovery": []
}

or

"python.testing.pytestRunArgs": [],
"python.testing.pytestDebugArgs": [],
"python.testing.pytestDiscoveryArgs": []

[github-actions]

Thanks for the feature request! We are going to give the community 60 days from when this issue was created to provide 7 👍 upvotes on the opening comment to gauge general interest in this idea. If there's enough upvotes then we will consider this feature request in our future planning. If there's unfortunately not enough upvotes then we will close this issue.

[eleanorjboyd]

Also related to #18778. Here is that whole comment from there:

I would like to use additional pytest options when running tests, but not when discovering tests.

Scenario

1. I want to generate coverage data on every test run, so I have this in my settings.json:

    "python.testing.pytestArgs": [
        "src/unittests",
        "--cov=.",
        "--cov-report=xml:coverage.xml",
        "-vvv"
    ],

2. But I don't want to generate coverage when debugging a test, so I disable it in my launch.json:

        {
            "name": "Debug Tests",
            "type": "python",
            "request": "test",
            "console": "integratedTerminal",
            "env": {"PYTEST_ADDOPTS": "--no-cov"},
            "justMyCode": false
        }

3. I also don't want coverage data when tests are discovered because discovery is way faster without these options. This seems to be impossible at the moment.

What I need

I think I need either a possibility to inject the --no-cov option somewhere around

vscode-python/pythonFiles/testing_tools/adapter/pytest/_discovery.py

Line 65 in e2bf90c

pytestargs.insert(0, "--collect-only")

or I need a possibility to just add the coverage options for non-debug test runs only.

[brettcannon]

Thank you to everyone who upvoted this issue! Since the community showed interest in this feature request we will leave this issue open as something to consider implementing at some point in the future.

We do encourage people to continue 👍 the first/opening comment as it helps us prioritize our work based on what the community seems to want the most.

[eleanorjboyd]

I created a proposal with a solution for this problem, would love your input on the above issue! #21845

[eleanorjboyd]

closing as the proposal will guide the solution creation to your PR