Filter NUnit Tree Display by Outcome

[CharliePoole]

As a design issue, we need to decide

• Level of selection: i.e. just success, failure, not run as we have been doing or should we support the full range of possible outcomes, including Inconclusive, Warning, Error? What about sub-levels?
• Hide vs dim unfiltered nodes in tree?
• What do we do if we have a fixture only failure and fixtures are not displayed? A namespace failure?
• How does the user activate the feature?
• What right clicks are available? What do they do? Corner cases, e.g. Right click and run failed fixture with some test failures and some successes under it. What happens?
• Do we do this one incrementally? I.e. simply first, more later?

[rowo360]

Here are some considerations regarding the outcome filtering (some of these applies also to duration/category filtering).

I believe we need to perform the outcome filtering ourself, because that's not a kind of filtering which NUnit supports natively. I imagine that the filtering is done on TestNode level and as a result of the filtering each TestNode has a bool property assigned (for example IsVisible). When building up the tree, this property can be evaluated to decide if a node should be created or not. Whenever the filter changes, the entire tree is build up once again (Strategy.Reload()).

Hide vs dim unfiltered nodes in tree?

I prefer to hide all filtered out nodes in the tree, so that I get a compact visualization of the tree displaying only those content which I'm currently interested in.

Generally speaking a TestNode will be visible if either it matches the filter itself or if any child matches the filter. So for example, if only all 'Passed' tests should be shown and a fixture contains failed and passed test cases, the fixture will be shown and all his passed test cases. And I would continue to show the fixture node as failed. That might be confusing on first sight, because I filtered for passed tests only. But the state of the fixture is nonetheless 'Failed' and starting to display anything different in the tree due to this filtering seems to increase complexity.

What do we do if we have a fixture only failure and fixtures are not displayed? A namespace failure?

Currently we don't start with showing/hiding fixture nodes. But I expect that we would show the namespace failure in this case because the outcome state is propagated to the parent nodes.

Level of selection

I propose to support the identical set of outcomes which is present as OutcomeGrouping:
"Failed", "Passed", "Ignored", "Inconclusive", "Skipped", "Not Run"
I didn't check it, but I expect that none of these require a special handling - but all of them can be handled identically. So there's no additional effort to support one more or less. What do you have in mind with sub-levels?

How does the user activate the feature?

I propose to add a new DropDownMenuItem which contains all outcome filters:

I would add a filter icon to this menu item to improve the UI and maybe it's sufficient to reduce the Text to 'Outcome'. And some additional UI gimmick idea: if a filter is set, the text and icon can be displayed in bold to indicate that this filter is active. Also the tooltip can already display the active outcome filters...

I had the impression that we need to add a new UI element class: a class similar to CheckedToolStripMenuGroup, but which supports multiple selected items (and not only a single one).

What right clicks are available?

I think that all 'run' commands must be adapted to consider only the filtered nodes. For example 'Run all' or 'Run selected': when executing these commands and some filter is active, we need to traverse the TestNode hierarchy and select only those TestNodes with IsVisible=True. So, we'll build up a TestSelection and pass it to the test run.
I prefer to keep the "View as XML..." command unchanged, so no filtering is applied here.

VisualState/Settings:
I don't know what is best here. I have the feeling there are arguments in all directions.
And if we cannot decide, it's best to not store the current filter at any location. Or do you have a clear preference?
So starting TestCentric the filter will be always be on default state. However when reloading or loading a new project I would keep the current filter unchanged.

Do we do this one incrementally

My idea is to divide this feature into these steps:

1. Add class representing a filter, provide filtering logic on TestNodes and evaluate 'IsVisible' when building up a tree
   So, although there's no UI to edit the filter, the filtering can be activate programmatically (in debugger). So, we can test the filtering already at this point of time, if it matches our expectations.
2. Add the filtering UI
3. Take care about the 'run' commands and ...

[CharliePoole]

Yes, we have to do this ourselves, NUnit supports filtering tests to be run. In this example we are actually filtering according to the test results after they are run. NUnit doesn't save the previous result in any useful way, so when running tests that failed, for example, we can only pass a list of tests to run.

On Hide vs Dim, we are in agreement. And all nodes have to show their actual outcome state not only to reduce complexity but because showing anything else would be incorrect. :-)

Outcome sub-levels refers to the fact that NUnit outcomes have a fixed list of statuses plus a label. The list you give includes a mix of both types, so it's probably better to deal with this in the process of implementation.

I'm not loving the dropdown menus that keep cropping up. Your images look a bit cluttered if you are intending for them to be always visible. I'd actually prefer filtering to be based on a new button above the tree. But maybe it's easier to deal with the visual side separately rather than as we implement each feature. Let's consider doing a whole set of non-functioning UI elements at some point. Meanwhile, for testing anything, I'm OK with whatever is easiest to implement.

I like a bottom up approach where we know what we want to do in advance and have drilled down to what is needed to implement it. The risk is that we end up over-engineering, trying to imagine in advance everything that any user may ever want to do. So let's not do that.

Along the same lines, I wrote all these questions down so we would have them as questions. Not because I think we should answer them all before we start coding. That's why I have been breaking out small bits of work that can actually be done right away as soon as I see them. If we were in the same place, all this would be on a white board and probably get erased every night. :-)

For this one, I think we are ready to start any time. But I think what's here is more than should be in a single PR, so consider how to divide it up. Maybe vertically, i.e. implement all the way through for a subset of what you want to do eventually.