Cancel test run doesn't seem to work

[rzhao271]

Testing #21494

I changed the parameterized test to go from 1 to 10k to 1 to 1k, and then pressed the run button, expecting the test changes to be picked up. However, I noticed the parameterized test was going past 1k because I had the tests expanded while the tests were running. I tried to cancel, but pressing cancel did nothing. I notice now that pressing cancel seems to work after the parameterized test finishes running. I'm wondering if the token can be checked per iteration of a parameterized test?

[bhavyaus]

Noticed the same behavior with vscode crashing after I hit Cancel Test Run in succession. I changed the parametrized test to go from 1 to 10k.

[eleanorjboyd]

On further testing with 10k tests I have found that the cancelation token is called immediately once the stop button is clicked followed by the TestRun object getting set to testRun.end(). I am not sure entirely about the delay, since I see one too and it seems to occur following the cancelation token call. When I observe it locally my extension logs print Setting the state of test "param_folder/test_parameterized.py::test_odd_even[93]" is a no-op after the run ends. for every test that was not finished and instead canceled. This error comes from here. I do notice that it spins for as long as it takes to print these logs which then stop spinning once it has gotten to the final one log but this is likely specific for the debugging mode I was in. @connor4312 do you have any thoughts?

This is at parity with the current testing infrastructure (rewrite vs non-rewrite code) so it is less urgent to get it out for this release but would be good to get to the bottom of. Thanks!

[connor4312]

It seems like the console logs are pretty slow, but there's also a UI stall when ending all those tests. I will have a PR for the latter, and the Python extension should avoid setting states after calling end()

[eleanorjboyd]

@connor4312 I thought we did avoid setting states after end() was called. Was there an instance you saw this was not the case?

[connor4312]

The console log "Setting the state of test "param_folder/test_parameterized.py::test_odd_even[93]" is a no-op after the run ends" is triggered by setting the state after end gets called. I also observed those logs when I was looking into the issue.

[eleanorjboyd]

Needs PR so that the Python extension should avoid setting states after calling end(). Check if this is accomplished by looking for "Setting the state of test "param_folder/test_parameterized.py::test_odd_even[93]" in console logs. This is a specific bug to the testing rewrite.

[eleanorjboyd]

should be fixed by: #21876

[eleanorjboyd]

think the edits I made should resolve this, closing so it can be verified

[rzhao271]

Reopening. It could be that the tests are cancelled but only after a long delay.

[pyth0nic]

Still an issue I think :(

[eleanorjboyd]

@pyth0nic, do you have a scenario where you are consistently see this? We are having trouble getting a good repro to test against.

[edubermejo]

@eleanorjboyd I am having the same issue. If I Cancel the Test Run it takes more than 2 minutes for the timer to stop and for the Run Tests button to reappear.

By creating an empty project and using pytest with a test like so:

import time


def test_slow():
    time.sleep(10)
    assert True == True

and clicking Cancel Test Run right after starting, the timer will stop after 10 secs have passed. Setting the time to any other number will lead to the run not being cancelled until such number of seconds has passed.

It seems the test run is actually not cancelled.

[eleanorjboyd]

closed by: #23262

also by work to move to using named pipes

[eleanorjboyd]

Verification:
follow raymond's original comment, let me know how cancelation is and if it feels reactive

[rzhao271]

It still seems to return the counter to 0/0 and keep the spinner going sometimes.
edit: it stopped at 93.2 seconds, which I can understand considering I changed the parameterized test to run from 1 to 10k, meaning the test runner was loading over 10k tests.
I'll consider the issue verified.

[eleanorjboyd]

@rzhao271 thank you! Yes I think it is just the time it takes to reset all the test items.