Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The Spin runtime tests use
conformance-tests
as a library (https://github.com/fermyon/spin/blob/967fdf368612478cef176bccc491faffad680050/tests/runtime.rs#L31). However,run_tests
entry point callslibmimic::Conclusion::exit()
, which causes the test process to exit at the end of the tests, with the result of thelibmimic
tests only. This means that if a Spin runtime test fails but all conformance tests succeed, the overall test suite still passes, disguising the test failure unless the avid reader pores over the logs.This PR proposes splitting the
run_tests
entry point intorun_tests_and_exit
, which retains the current behaviour, andrun_tests_to_conclusion
, which instead returns the Conclusion object.libmimic
still prints all progress and results as normal.With this in place, the Spin runtime test that checks conformance will be able to examine
Conclusion::has_failed
and propagate failure (e.g. by returning an error such as "One or more conformance tests failed") or pass on success, without interfering with other tests.Of course the names and behaviour are totally up for grabs (some names are deliberate to make consumers aware of changed behaviour, but that may be misguided), and if you feel there's a better way of managing this then please feel free to circular-file this - I'm not at all familiar with the code base or with other use cases I'm afraid.