Add differences from NUnit V2 GUI to documentation

[maettu-this]

Describe the bug
Tools > Settings... > Gui > Project Editor mentions the NUnit Project Editor (which I guess is https://github.com/nunit-legacy/nunit-project-editor, but that has seen no update/commit for the last 7 years...) but File > Open... does not seem to foresee opening anything but an assembly. Trying to open an "old" NUnit 2.6.4 .nunit file results in a red cross.

Attempting to upgrade the existing YAT.nunit from NUnit 2.6.4 to NUnit 3.x and then NUnit 4.x within YAT FR 293. Struggling to get TestCentric handling the 16 referenced test assemblies in a convenient way. Having to manually File > Add Test File... is a no-go, especially given the set isn't perserved on the next launch (sounds related to #211 and #3), and given there seems no way to switch from Debug to Release (sounds related to #785).

If TestCentric doesn't intend to provide these handy NUnit 2.x GUI features, I think the above mentioned issues should comment on this and Tools > Settings... > Gui > Project Editor should be removed, in order to not confuse/mislead the user.

To Reproduce (not needed to get the root of this issue)

1. SVN get https://sourceforge.net/p/y-a-terminal/code/HEAD/tree/trunk/.
2. Upgrade https://sourceforge.net/p/y-a-terminal/code/HEAD/tree/trunk/YAT/NUnit/Binaries/ to NUnit 3.14.0.
3. Try to get TestCentric open and persist on YAT.nunit.

Expected ideal behavior
TestCentric should support existing NUnit projects out of the box.

Expected minimum deliverable
Better guidance on how to handle multiple test assemblies and multiple configurations (e.g. Debug and Release). https://test-centric.org/testcentric-runner/ should state whether and how this can be achieved. The documentation should also state the features of NUnit 2.x GUI yet missing in TestCentric, or not intended to ever been implemented.

Screenshots
N/A

Environment (please complete the following information):

• TestCentric Version 1.6.4
• OS: Win10 Home 22/H2
• Target Runtime for your test assembly: .NET Framework 4.8

[CharliePoole]

There is nothing to do to "convert" an NUnit project from an old version of NUnit to a newer one. The format of the .nunit file has not changed and the framework doesn't read the files anyway. That's why the legacy nunit-project-editor can still be used for them. I was able to install the editor just now from chocolatey and open your YAT.nunit file with it.

The intention of Tools > Settings... > Gui > Project Editor is to allow the user to set whatever editor is desired as the one to use when editing project files. It could be the nunit project editor or it could be notepad. So I don't think we should remove that setting. It may need a better explanation in the dialog itself however.

TestCentric doesn't know how to load projects any more than NUnit GUI did. Both of them use extensions for that purpose. It's possible that you used a package for the NUnit GUI, which bundled a number of extensions. TestCentric does lest bundling and requires you to install most extensions you might want to use. That's trivial if you use chocolatey but a bit more work using other types of packages. We'll probably add a menu item to the GUI to do that eventually.

BTW, issues #211 and #3 are not related to NUnit projects. #211 mentions them but further along it's decided to just save the test package, which may have one or more nunit projects included in it. However, the GUI already opens and uses projects for loading tests. Work stopped on 1.6.4 precisely because it already does everything that the NUnit GUI did. Open issues all appply to TestCentric 2.0.

I think it would be helpful if you could back up a step and explain what you are trying to do. It seems as if you are doing two or three conversions at the same time, which sounds confusing - or even scary. :-)

Is there a reason you are using 1.6.4 rather than a newer version of TestCentric?

[maettu-this]

Hmm, seems like a major mismatch or misunderstanding. Let me detail one step at a time:

TestCentric doesn't know how to load projects any more than NUnit GUI did

But the 2.6.4 NUnit GUI did offer to open .nunit files out-of-the-box:

The \addins folder is empty and there is nothing indicating a bundle installed:

Also did that version offer to select Debug or Release out-of-the-box:

TestCentric neither has a Project menu nor File > Open Project... nor Tools > Settings... > IDE Support > Visual Studio which would allow using a .sln file.

using 1.6.4 rather than a newer version of TestCentric

There is no particular reason. Tried both, haven't managed to conveniently open multiple assemblies (i.e. some form of a "project") on either 1.6.4 nor 2.x beta.

back up a step and explain

I am trying to upgrade my project from NUnit 2.6.4 to the latest 4.x version, taking 3.14.0 as an intermediate step. CI is one thing, being able to run the tests manually is the other. And since NUnit 3.x/4.x no longer offers a GUI, I am trying to use TestCentric for that purpose. It's as simple as that.

[CharliePoole]

Summarizing...

2.6.4 running projects out of the box

You seem to be correct. At some point we stopped doing that without an extension but I must not have remembered correctly when it happened. Possibly in the lead-up to 3.0. I'll check on GitHub.

2.6.4 running allowing selection of config for a project

Yes, that's correct. It's what issue #785 is about. Of course, that will only be fixed in the 2.0 release, not in 1.6.4.

Project menu... File open menu...

There's no project menu in the GUI because it doesn't have the functions that would go into a project menu. NUnit GUI had one because NUnit V2 treated all the files you opened as a project... that is, if you opened multiple assemblies, it became an anonymous nunit project, which you could later save under some name.

File open is what version 2 of the GUI uses to open NUnit and VS projects... provided you install the correct extensions with it.

Upgrading to NUnit 3 and then 4

That makes sense to me. Note that the nunit-legacy releases after 2.6.4 were intended to make that easier in certain circumstances. I created them after the majority of the team members voted to end support of version 2. Whether those features are useful in your case depends on your code so read the release notes. BTW, 2.7.1 was the last release. There won't be any more. So if you use it consider it as a temporary measure only.

SUMMING UP...

I would suggest using the latest production release of the NUnit console runner in your CI for either NUnit 3 or NUnit 4. The latest beta release of TestCentric should work for desktop use but consider also converting your tests to use NUnitLite. The Console or Gui can still run those test but you can also run the test assemblies one at a time on the desktop. Some developers like doing it that way.

As a workaround until issue #785 is fixed, I suggest having two different project files, one for debug and one for release. Just run the one you are working on at the moment. A couple of desktop shortcuts make that trivial or just use the GUI menu to open the one you want.

Let me know if you think I can close this as a duplicate of #785 or if you think something more should be done.

[maettu-this]

Charlie, thanks a lot for the elaborate summary. I confirm that upgrading to 2.7.1 as a first intermediate step has worked fine, already having migrated to Assert.That several years ago didn't need much more migration except for renaming TestFixture... to OneTime....

Note that 2.7.1 still provides the project functionality mentioned previously.

provided you install the correct extensions with it

The "correct" is what makes it non-obvious to me. I guess for you being deep into NUnit this is a no-brainer, but for me being a "simple" NUnit user (actually I use NUnit in a quite complex test generation setup) neither the TC extension dialog documentation nor TC @ NuGet nor TC @ Chocolatey sheds any light into what the "correct" extension to achieve NUnit 2.x GUI project behavior could be.

consider also converting your tests to use NUnitLite

This could be a good way forward for the basic unit tests of my project. For the higher level systems tests that require file system and I/O access NUnit however seems the more appropriate choice, given they currently require quite some test fixture setup and teardown.

[CharliePoole]

Well, it's in the nature of extensions that the code being extended doesn't know what the extension does. :-)

So the documentation of what an extension does has to be found in the extension project. That said, the online NUnit documentation does describe what types of extensions exist and what extensions are available from the NUnit project.

However, to simplify, you need the NUnit Project Loader extension to load NUnit projects and the Visual Studio Project Loader extension to load VS projects. :-) Note that both of these extensions will work with whatever version of the NUnit framework you are using.

I highly recommend using chocolatey to install both the test runners (console and GUI) and the externsions, since it puts them all in one place and really simplifies use of extensions. I'd only use nuget packages within each project if you wanted to use different extension versions for different projects.

[maettu-this]

Thanks for the hint on choco. I fully agree with you, choco feels more natural for tools and extensions than NuGet.

the online NUnit documentation does describe what types of extensions exist

To be honest, for me as a user this is not obvious at all. Having installed TestCentric, looked at the menus, read through the documentation, how should I conclude that I have to check the NUnit documentation for loading extensions into TestCentric? I'm sure there are very good reasons for this separation and the interfacing APIs. But coming from an out-of-the-box user experience with 2.6.4, having to install three pieces of software and still not getting the same user experience looks like a step backwards.

Maybe the TestCentric extension settings documentation could hint on where to look for extensions, and for what sort of extensions to search for. And maybe the TestCentric documentation could describe what to do for getting an NUnit 2.x-like user experience. This is about what the initial intention of this very issue was.

[CharliePoole]

Yes, I do see your point.

The logic behind not documenting installation of the NUnit extensions is that they are, after all, NUnit extensions. They are treated as third-party extensions. In fact, NUnit itself only documents the extensions written by the NUnit team.

That said, it does make sense to list the NUnit extensions we support in the docs and to give some information about how to install both types of extensions. I'm marking this as a documentation issue for that purpose..

[maettu-this]

Yes there really is the need for more documentation, or documentation at additional/other locations. Here is what I have tried:

1. Installed the NUnit Project Loader Extension 3.7.1 by choco install nunit-extension-nunit-v2-result-writer, wondering why "Project Loader" is named "result-writer" on choco and NuGet.
   • The Visual Studio Project Loader Extension stated "load solutions and projects in Visual Studio format" sound more like a "Project Loader", but that doesn't include .nunit files.
2. Upgraded my tests to NUnit 3.14.0 (another intermediate step towards 4.x).
3. Tried to use TestCentric 1.6.4 on that, using the existing YAT.nunit with command line C:\ProgramData\chocolatey\lib\testcentric-gui\tools\testcentric.exe YAT.nunit.

No longer having a Tools > Exception Details... menu doesn't let me get the details for the issue.

4. Upgraded to TestCentric 2.0.0-beta4 choco upgrade testcentric-gui -pre

I understood the TestCentric GUI being created as an upgrade path from the NUnit 2.x GUI. Being stuck again let's me pondering...

[CharliePoole]

First, let's be sure you know where the existing documentation is...

NUnit has pretty good docs for its extensions: https://docs.nunit.org/articles/nunit-engine/extensions/Installing-Extensions.html. Notice the phrase "when both engine and extensions are installed from the same package manager, all extensions are installed automatically." This could be a little misleading and should really say "all extensions, which you have installed, are found automatically. So long as you install both the runner and any extensions using chocolatey the runner will find the extensions.

As you know, TestCentric extensions are documented here: https://test-centric.org/testcentric-runner. The description of the engine has a TODO item, which we will complete via resolution of this issue. Of course, the only extensions documented are the pluggable agents, which are incompatible with the NUnit engine. So, also as a part of this issue, I will be adding an explanation of how the TC engine is backwards compatitible with NUnit as regards extensions. That is, it loads both "TestCentric.Extension." and "NUnit.Extension." extensions.

Now, for going forward...

As part of this issue, I'll be updating docs for version 1.6.4 to clarify what works and what does not. That includes, which versions of the nunit extensions are supported. Unfortunately, the test code for 1.6.4 didn't freeze the extension versions it uses and some new versions have been released since that time. In any case, I won't be fixing any of these problems or releasing a 1.6.5 version.

For TestCentric 2.0 beta 5, I plan on fixing the issue #785 and verifying that it works with the latest releases of all NUnit extensions OR at least documenting which versions it supports. This issue will add additional documentation to beta 5 as well.

For you...

When I actually know which versions of the extensions work with TC 1.6.4, I can post here. However, it really makes no sense to me that you continue to try to run under 1.6.4. Since you haven't used TC before, there is no backward compatibility issue. Why not just work with 2.0 beta 4 from the start?

If you have already converted to NUnit 3, then you should be able to run your test assemblies under that beta. So make sure that is working for you first. If you want to run NUnit 2 tests, then you need the chocolatey package nunit-extension-v2-driver.

Only add a single extension at a time because of issue #989. This has been a problem for a while so I'll try to get it into the next beta as well. Meanwhile, make sure each extension doesn't cause a crash before adding a new one.

Once you can run test assemblies, I suggest you then install the latest nunit-extension-nunit-project-loader and use the workaround of two different projects for Debug and Release until #785 is fixed.

[maettu-this]

We'll eventually get there..

Note that part of the confusion comes from https://github.com/nunit/nunit-project-loader/releases:

For a chocolatey install: nunit-extension-nunit-v2-result-writer.3.7.1.nupkg
For a nuget install: NUnit.Extension.NUnitV2ResultWriter.3.7.1.nupkg

And:

You may also download the extension from chocolatey.org or nuget.org.

Isn't this simply wrong? The artifacts further down are correctly named nunit-project-loader.

Having tried TC 1.6.4 first because auf 2.x being beta and I found an empty menu entry, indicating it indeed is beta:

I tend to have an "aura" to often run across "strange" issues. Whenever I can I therefore try to avoid immature looking things. They have distracted me too often. Pretty sure you know what I mean.

I have completely removed TC and again started from scratch:

1. choco install testcentric-gui -pre
2. Successfully ran a single test assembly:

3. choco install nunit-extension-nunit-project-loader
4. Tried to open TC again:

5. choco uninstall nunit-extension-nunit-project-loader
6. Single test assembly fine again

[CharliePoole]

Yes that's just a textual error. I added a comment about it to a related issue for that project.

I'll test further myself to see which version - if any - of the project loader actually works with TestCentric. You could try the version before the last one to see.

[maettu-this]

Both 3.6.0 and 3.5.0 result in the above shown error message.

[maettu-this]

nunit-extension-vs-project-loader 3.9.0 and 3.5.0 also result in the above shown error message.

[maettu-this]

TestCentric 1.6.4 works with the nunit-extension-nunit-project-loader :-)

Finally found a setup that could be good enough for the moment. Trying to run the full test set on my project and see what needs to be adjusted to become compatible with NUnit 3.14.0 and TestCentric 1.6.4. If all works out well, I will use NUnit 3.14.0 and TestCentric 1.6.4 for a while.

Thanks very much for your support @CharliePoole! Hope my feedback lets you make TestCentric even better!

[CharliePoole]

That's great. Can you tell me what worked for you?

Thanks for the feedback as well.

[maettu-this]

Took me a while to adapt my .config based test case generation to the AppDomain related changes in NUnit.

(By the way, how does NUnit deal with AppDomain functionality no longer available in .NET 5+? Asking because my project also uses multiple AppDomain for separating user defined scripts from the core application. Migration to AssemblyLoadContext has the potential to require quite some efforts and redesign.)

Back to TestCentric. In terms of...

• .nunit file based project loading
• .config file based test generation for physically available devices
• category based test include/exclude
• running tests

...TestCentric 1.6.4 with the latest nunit-extension-nunit-project-loader 3.7.1 works fine.

However, came across #1043 which much limits debugging of long running tests against stateful testees. (My project deals with physical devices, which by nature may be long running. And my project implements stress and endurance tests which may run for hours and then reveal an issue. Not having synch'd Trace and Progress output much limits analysis of such test runs.)

(By the way, having read through to whole list of release notes since NUnit 2.6.4 (https://docs.nunit.org/articles/nunit/release-notes/framework.html and https://docs.nunit.org/articles/nunit/release-notes/Pre-3.5-Release-Notes.html) it is very impressive to see what NUnit all covers!)

[CharliePoole]

Glad it's working for this stage anyway. I suggest continuing on your upgrade path by upgrading only one thing at a time. :-)

We'll keep this issue for TC 1.6.4 documentation. Let's do new issues if you run into anything when you switch to TC 2.x.

Regarding NUnit's use of AppDomain:

NUnit 3.0 stopped using appdomains for .NET Core and .NET 5.0 targets. You will have to use process isolation for that purpose. The 4.x code for the console runner drops all user control of appdomains and processes - at least direct control. You have to specify a particular agent instead. However, 4.0 has not yet been released so you are good for now.

Regarding framework changes over the years:

Thanks. Yes, we did a lot. On the down side, however, what was once a very simple framework is no longer so simple. :-(

Finally... I wanted to tell (or remind) you that while using the nunit-console (as well as TC GUI 1.6.4) you may create an engine version conflict if you also reference the NUnit 3 VS Adapter. Each of these brings in a particular version of the engine and you need to know what that version is and ensure you are not trying to load two different versions.

This post is a bit out of date but at least explains the problem. Note that it should not be a problem with GUI 2.x, which no longer uses the NUnit engine.

[maettu-this]

Thanks for the hint on the potential engine version conflict, I'll keep that in mind.

[CharliePoole]

Renaming issue to reflect that we are dealing only with TC GUI version 1 documentation here.

I'll update the release notes through 1.6.4 (currently only through 1.6.2 on the website) and add a new page to describe differences from NUnit V2.

[CharliePoole]

@maettu-this Please review https://test-centric.org/testcentric-gui-version1/nunitv2-comparison and let me know if I missed anything.

[maettu-this]

Content:
Perfect.

Title:
"Nunitv2 Comparison" should better be "NUnit V2 GUI Comparison".

Location:
Doesn't this also apply the TC 2.x? The GUI does look different, but the menu including [Tools > Settings... > Text Output] still resembles the NUnit 2.x GUI. In my opinion, the documentation should also be available for TC 2.x.

Other than that, feel free to close this issue.

[CharliePoole]

Thanks @maettu-this ,

I'll fix the title. However, I do think it belongs in the V1 docs only.

While TC v1 and v2 are both completely new programs, v1 is deliberately made to be similar to the NUnit GUI. It's only ew feature is that it can run NUnit 3 tests.

V2, otoh, is entirely new in terms of both how it looks and how it behaves. It works differently from NUnit's GUI in several ways. At some point, there might be a need for a guide for the use of former NUnit users, but I think it would be more in the scope of a long article or book chapter than a single page. I think you'll see what I mean when you are able to get into using V2 yourself, but you have probably endured enough change for the time being. :-)