How should plugins perform a "final" test?

[Krinkle]

Following from #1629, the ember-cli-memory-leak-detector plugin used QUnit.done() to register a final test.

This no longer works as of #1629. In that PR, I stated that this was already unreliable and could be ignored silently without realizing. But, it's also quite possible that it happened to work reliably for all cases that mattered for the specific way in which ember-cli and Testem aggregate test results.

This ticket is to determine (and make if needed) a viable way for plugins to register a "final" test, and to make sure it is documented going forward.

@steveszc wrote:

[it] is breaking some functionality in https://github.com/steveszc/ember-cli-memory-leak-detector

My use-case is that I need to inject a test after all other tests have finished running (once the tests finish I capture a heap snapshot, search it for memory leaks created during the test run, and create a new test that either passes or fails depending on whether memory leaks were found).

I previously relied on the (apparently unintentional) ability to create a test after the test suite has finished. I understand that I have a rather unconventional use-case but it would great if I can continue reporting memory leaks as a failed test within Qunit as it provides a great DX.

Is there a way to do this moving forward?

[Krinkle]

@steveszc There are two broad things that come to mind.

Firstly, I'd recommend trying to define this test module from QUnit.begin(). This may seem counter-intuitive, as this callback is named for running before test execution and user interface construction (i.e. the start of the run). But, in relation to loading the tests, it aligns well with your use case. The begin callback is run after all test suites have been loaded and all tests have been defined, just before the execution begins. It allows plugins like yours to define additional tests which are naturally placed at the end of the queue.

Secondly, this might be a good use case for allowing plugins to register an "after" hook with a hypthetical global module, which would be an ancestor to all modules and tests and thus would fire once toward the end of the test run, but before it closes the result for QUnit.done() and CI reporting, etc. This is not currently exposed to plugins, but there is an issue for it at #1475, which we could certainly prioritise for this.

[Krinkle]

I'm closing this per my advice at steveszc/ember-cli-memory-leak-detector#49 (comment). If that doesn't work out or if there's any way I can help, feel free to comment below (I'll re-open), or file a new issue any time!