Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Port RandomizedTesting Test Runner #122

Open
bongohrtech opened this issue Oct 10, 2019 · 3 comments
Open

Port RandomizedTesting Test Runner #122

bongohrtech opened this issue Oct 10, 2019 · 3 comments

Comments

@bongohrtech
Copy link
Owner

While the test framework can run with the standard NUnit test runner and we have managed to make it work, it is missing the ability to reproduce a random test failure, which is crucial for debugging some of the more complicated random tests. Lucene uses a custom JUnit test runner called RandomizedTesting to accomplish this.

It also includes some other nice features

  1. Ensure when a test fails, the random seeds are included in error messages and logs
  2. Code analysis to ensure the tests are setup properly
  3. Run tests in a random order to ensure they are not dependent upon each other

Preliminary analysis shows that the API for NUnit allows building custom test runners and it is a close enough match to implement the functionality. Most likely, this will require a custom adapter as well, so the test runner can integrate into Visual Studio and dotnet test/mstest.

Do note that without doing some pretty heavy refactoring on the Codec, Directory, and LuceneTestCase classes, it is not possible to run tests in parallel within the same AppDomain because the codecs use a static variable to turn codec impersonation on/off. For now, it would probably be best to run tests serially.

JIRA link - [LUCENENET-627] created by nightowl888
@bongohrtech
Copy link
Owner Author

I have opened a dialog with the NUnit team, trying to gauge whether we can get them to add the most important features we need, whether it will be entirely up to us to port the test runner and create our own Visual Studio / dotnet test adapters, or somewhere in the middle: nunit/nunit#3405

by nightowl888

@bongohrtech
Copy link
Owner Author

After updating the test framework to utilize the NUnit.Framework.TestContext.CurrentContext.Random property, I noticed that random test failures now repeat reliably until the next build. Whether or not there is a way to lock in that behavior from one build to the next, this is probably adequate enough to avoid porting the entire RandomizedTesting framework.

That said, it would probably make sense to create a new library with the random generators so they can be shared without a reference to our test framework and all of the dependencies that come with it. We should also migrate the extension methods for the Random class.

by nightowl888

@bongohrtech
Copy link
Owner Author

Lowering priority of this and removing the 4.8.0 version, as it is not required for the Lucene.NET release. It might still be useful to do this someday, so I am keeping the issue open.

by nightowl888

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant