2.0.0-beta6 force stop endless loop test - cleanup

[MartinZec]

Hello,

I forced to stop a endless loop in a test.
After it is sopped, I can execute the test again but I cannot stop it anymore because the stop button is grayed out.
In case I close the GUI, it tells me that a test is currently running.

The endless while loop is also preventing logs.
In case endless loop is removed logs are shown at text output:
"=> name1.name2.name3_1.name3_1.test1
test1test2test3"

Start and end time are wrong displayed in case of a forced stop.

Can be reproduced with attached example project.
Solution1.zip

Best regards,
Martin

[rowo360]

Thank you for reporting this issue and providing a sample project. So I was able to reproduce this problem easily!
And of course this is not the desired behavior.

After starting the first test run, the toolbar looks like this - all the 'Run' buttons are disabled and the 'Stop' button is the only activated one.

So, I first clicked 'Stop' and afterwards it changed to 'Kill' which I clicked also.

After starting the second test run, the toolbar looks like this - indeed the 'Stop' button is disabled, but also all the 'Run' buttons are still enabled.

So I have the impression that we fail to start the second test run at all, but still struggeling to stop the first test run properly.
May I just ask you to confirm that the toolbar looks identically to the 2nd screenshot on your side? Thanks!

Here are some additional observations which might be helpful for further analyzing this issue:

• On my computer the endless loop generates an OutOfMemory exception after ~25s - don't know exactly why. But removing the Console.Writeline(...) solves it.
• I also got the message 'A test is running' while closing the GUI - however not always. If I sometimes operated a little slower, then the 'Unloading' message appeared for a while and finally the GUI close.
• I noticed that the process "Net462 Pluggable agent" is terminated properly when clicking the 'Kill' button. But no new process is started for the second test run.

[CharliePoole]

Normal stop (Stop button) is passed to the agent and it's actual test runner. It gets the message and stops staging any more tests. Then it waits for all running tests to finish. A test in an infinite loop, of course, never finishes. After waiting a while, it gives up.

The GUI, not receiving a notification that the run has stopped, displays the kill button. Clicking it causes a forced stop, which is also passed to the agent. The agent then exits the process, never sending a notification.

So there are several problems here...

1. No notification is sent.
2. The GUI doesn't wait for a notification but just assumes the Kill button worked and the run terminated.
3. More subtly, the agent is making no attempt to kill the thread on which the test is running. That would be possible for a .NET Framework agent but not for .NET Core.
4. We have tests of an infinite loop in the GUI, but they are not pointing out this problem to us.

@rowo360 This is probably multiple issues. So far you have only worked in the GUI (item 2) but I'm happy to support you if you want to explore these new areas. I'm also happy to take on any pieces of this.

Items 1 and 3 probably reside in TestCentric.Agent.Core, the common code for all agents.
Item 2 and 4 are in the GUI but the tests should probably be in the engine and/or the the common agent code as well.

[CharliePoole]

@MartinZec Was this a problem in the beta5 release?

@rowo360 It would also be useful if we could identify an easy temporary fix for the GUI, while waiting for anything to be done in the other projects, However, I wouldn't be opposed to doing it all and making a new beta release.

[MartinZec]

@CharliePoole I did not check the beta5 release up to now. I just downgraded beta6 to beta5 and see the same behavior there.
@rowo360 Yes, the toolbar look exactly the same as in your screenshots.

[CharliePoole]

@rowo360
Let's make this issue about items 2 and 4 in my comments above. I'll create an issue for items 1 and 3 under TestCentric.Agent.core. I assigned it to you.

Todo...

• Figure out how to test this. We have an assembly, hung-tests.dll, that was intended to be used for testing. But the gui has no unit or package tests that make use of it. You should be able to run it manually from the GUI and see if stop still works. Assuming it does, then there's something unique about this test case, maybe due to SpecFlow.
• Try waiting for a termination notice after we try to kill the run. If I'm correct, then this bug will be replaced by a hang, which is obviously bad. But that will be fixed if TestCentric.Agent.Core sends a notice.

This issue will need to stay in process until TestCentric.Agent.Core is fixed and all agents are updated to use that fix. I'm making it a low priority for that reason, although the issue itself is quite important. I'll bump up the priority after those other fixes are in.

[CharliePoole]

@rowo360
Well darn it! I created an issue for this in TestCentric.Agent.Core and started working it. What I find is that the engine is actually generating an end of run event. So the GUI seems to be acting correctly. That is, it gets the signal that the run is complete.

Can you make this run correctly by changing anything in the example project?

[rowo360]

I'm back from vacation and continued to work on this issue:
First of all I'm able to reproduce this issue also with our own test assembly hung-tests.dll

User point of view:
I believe the stop command works properly: when the user hits this button, a message is shown and as soon as the current test run is finished, the user can start new test runs. All of this works fine!
However the kill command does not work completely. And that's independent if we are dealing with an endless running test or a regular running test. When the user hits the 'kill' button, all toolbar buttons are getting enabled again, so that the user can start a new test run. So, he might get the impression that the 'kill' succeeds, however he cannot start any new test run. Whenever he hits the 'run' button, nothing happens.

Technical point of view:
When debugging this scenario, I noticed that the 'Kill' actually works correctly. Indeed the process executing the tests is terminated and an event is fired, so that the GUI updates to the new state. However the problem is that the 'kill' leaves the involved classes in a state so that they don't manage to start another test run. And we fail to recover from this inconsistent state.
I prepared a fix in the TestCentric.Engine project and will create a PR for it.
(Here's the link to the PR 213 )