Issue Revisited: Console Output

[NightOwl888]

Following on to this comment about the Breaking Change between NUnit 2.6.3 and NUnit 3.5.0.

I am also experiencing this issue. Previously, writing Console.WriteLine would make the statements appear in the Output/Tests window. The "solution" to "keeping the tests in context" using the test output window has one huge limitation - the length of text that is allowed there.

We have two settings in our application (that is ported from Java) - silent and verbose. The verbose setting prints a LOT of information, some of which is useful for debugging. Nearly ALL of it is truncated from the Output link under the specific test (all that appears is the header information that tells the configuration, the useful debug information is gone).

Okay, so I can copy and paste it EVERY TIME I RUN IT into a text editor, but that is a huge productivity loss.

Unfortunately, because we are now supporting .NET Standard, there is no way to downgrade back to NUnit 2.6.3. So, other than attempting to write the debug info from the unit tests into very chatty Output/Debug window and then having to sift through it, is there any kind of hack/workaround we can apply to print to the Output/Tests window where the debug info is preserved without being truncated so much that it is entirely useless?

[FransBouma]

I know I'm late to the party here, but maybe this will help:

For v3, trace output isn't picked up, as I found out the hard way as well. After a day fighting with this (also on .NET core 2.0), it appears to be rather straightforward.

As NUnit stores all output send to console output with the test result, a single tracelistener is enough:
Trace.Listeners.Add(new ConsoleTraceListener());
This will store any trace info resulting from a test with the test results for that test, and if you select the test in your runner, e.g. R#, it will show the output there.
For .NET core 2.0, you need to use:
Trace.Listeners.Add(new TextWriterTraceListener(Console.Out));

Do this one time, in the OneTimeSetUp method in your TestFixture.

It doesn't end there though :) You can then add another tracelistener and write to a file.
Trace.Listeners.Add(new TextWriterTraceListener(@"c:\temp\testoutput.txt"));
add that line before or after the consoletracelistener and it writes the output also to that file.

To make sure the data is written to the file, add:

		[OneTimeTearDown]
		public void TearDown()
		{
			Trace.Flush();
		}

so it's flushed at the end of the tests.

Again, it's late, you likely already have moved on but perhaps others like me who run into the problem of not having any trace output in nunit 3 can again sleep well :) (I moved to nunit3 because we too had to support netstandard2.0 so was confronted with this).

A redundant note perhaps: parallel tests likely won't work with this.

[psulek]

@FransBouma I'm currently experiencing same issue. I have netcoreapp1.1, xunit. Trying your workaroud with add trace listener and flush does not help me. Resharper unit test output is still empty. Can someone point some tips redirect dotnet test command output into resharper unit test output window?

[chostelet]

Hello,

I concurre, having the same kind of issues. Debug.Print() statements in the application are no longer being visible in the Output window.
And, as suggested above, being obliged to add a [OneTimeSetup] method just to add a
' Trace.Listeners.Add(new ConsoleTraceListener());' is hardly manageable when having thousands of unit tests.
And even the result (output) is no available at once for the whole tests playlist, but needs to be pickup in each test.
Hard to believe.
And this should be documented in the Known Problems!
Regards

[CharliePoole]

Let's note we are talking about two entirely different things here. @NightOwl888 started this issue about capture of Console.WriteLine. @FransBouma and subsequent commenters have been talking about Debug and Trace output.

It might seem as if they are the same, provided you are directing your trace to the console, but it isn't. NUnit intercepts your Console writes and the original bug applies to that. NUnit does absolutely nothing with Debug and Trace output. So we may need to look at these two things separately.

[chostelet]

It looks like, the net result to the user appears to be the same => No output in a straightforward way.
I understand having the capability of parallel execution can mess up the output, but this is something one should expect anyway.

[CharliePoole]

@chostelet It's definitely an issue for the user but I wanted to note that fixing the one (console output) might not necessarily fix the other (debug / Trace). It's info for whomever takes on the issue.

[chostelet]

Hi,
You're right.
As a temporary workaround, does solving the console output may help get the default debug/trace output?

[CharliePoole]

Not impossible. 😄

[NightOwl888]

BTW - a side effect (bug) from writing all of the output for each test into its own buffer is that the buffer is set to some fixed size somewhere, and if you exceed it the test runner fatally crashes.

For now I have isolated the handful of tests where it happens and cut down the amount of testing that occurs when verbosity is enabled and/or manually hard coded them to not write output in order to allow the test runner to keep going. But is there a setting somewhere to increase the size of the buffer?

[CharliePoole]

NUnit simply appends the output for each test to a string. Some buffering may be happening before we get the output, depending on which output source is involved.

[mipnw]

One problem with making Console output not available in the console but available elsewhere, such as captured and made available via Visual Studio Test Explorer + click on test + click on "Output" is that not everyone is using Visual Studio to run NUnit tests. Visual Studio Community 2017 as of this writing is only available for Windows and MacOS, not Linux.

What about people who run their unit tests on the command line, on a headless machine perhaps, or one day on Linux, the next day on Windows. Sometimes the test machines change so fast you don't have time to install Visual Studio you just copy the tests. All environments have a command line. Sure there's ways to configure VS to do remote debugging, and remote testing, but again, that's if you use VS.

It'd be nice if something like "dotnet test" printed everything the test printed. Sure NUnit may warn us ordering might be wrong, because of parallelism, a bad order is still better than no output.

Then there's the counter argument I sometimes hear, that unit tests should just pass/fail and not print anything because a human shouldn't have to read test output for any "good" test. That's a decision the test designer ought to have, not the test framework. Sometimes a unit test is just a fast way to call a library function during development, iterate based on console output, until the test designer reaches that stage where it's simply pass|fail. The development phase where the test prints something is a valid scenario that probably helped enhance the final test. Possibly console output saved back as a constant in the test and used as an Assertion, when writing that constant by hand would have been particularly tedious.

[CharliePoole]

To be clear... NUnit itself doesn't prevent the output from appearing at the console. By "NUnit" I mean the NUnit Framework, which runs the tests. It does capture the output written by your test and passes it to whatever runner has invoked the tests - nunit-console, vs adapter, etc. That runner has the choice of where to display the output.

In the case of dotnet test at the command-line, you are hampered by the fact that it uses the same adapter that is used when running within the IDE. IF (that's a big if) dotnet test were using some other module, specifically intended for command-line use, then it could do what the nunit3-console runner already does: just display the output or even pretty it up by adding the name of each test along with it's output.

Alternatively, the vs-adapter could somehow detect the environment under which it is running and do something different when run by dotnet test. Generally, however, I prefer two programs, perhaps with some common code, to one program that does two things. 😄

[mipnw]

Thanks. I hadn't paid enough attention to the distinction between unittest framework and runner.

Do you know if referencing the NUnit.ConsoleRunner package inside the test project instead of referencing the NUnit3TestAdapter package would help with running tests via dotnet test ?

It does not out-of-the box. When I change the runner and rebuild, dotnet test says "No test is available in ProjectName.Tests.dll Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again."

Since I've never used that runner I may be missing something obvious, or I may be misunderstanding what that runner is entirely. Looking for NUnit.ConsoleRunner documentation led me to https://github.com/nunit/docs/wiki/Console-Runner, which talks about an exe program called nunit3-console.exe, not quite a package you can reference in your own test projects. I'm confused.

EDIT:

Also found https://github.com/nunit/docs/wiki/.NET-Core-and-.NET-Standard which says "Testing .NET Core and .NET Standard projects requires each test project to reference version 3.9.0 or later of the NUnit 3 Visual Studio Test Adapter.", so using NUnit.ConsoleRunner may not work.

[CharliePoole]

The docs you found for nunit3-console.exe are what we mean by the "NUnit console runner." It can be installed into a project using the NUnit.ConsoleRunner package and is then available to run from your packages directory. I find it handier to use chocolatey, which adds it to my path. The chocolatey package for the console runner is nunit-consolerunner.

However, the console runner unfortunately only supports .NET Framework at this point, not .NET Core. When working with .NET Core, you must use the adapter as you found out.

[OsirisTerje]

The trace/debug is moved to #718. Currently we don't get information from NUnit without adding a tracelistener, as @FransBouma states above. The tracelistener can be added in multiple ways inside an assembly, so that is probably a good way. With 3.17 Console Output is working, so using the Tracelistener is probably the best way to handle this for now.