-
Notifications
You must be signed in to change notification settings - Fork 14
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: add a randomizer proxy to test (some part of) the testsuite #29
Draft
laurentsenta
wants to merge
12
commits into
main
Choose a base branch
from
feat/randomizer-test
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
laurentsenta
changed the title
feat: add a randomizer test
feat: add a randomizer proxy to test (some part of) the tests
Mar 31, 2023
laurentsenta
changed the title
feat: add a randomizer proxy to test (some part of) the tests
feat: add a randomizer proxy to test (some part of) the testsuite
Mar 31, 2023
I like it! That's really clever ❤️ |
Maybe we could run tests against random gateway a couple of times. Or rerun the tests that passed. Then, if we found a test case that passed on all the attempts, we'd know something's wrong. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The worst thing that could happen to us is introducing a bug that disables one or more checks.
An example: in Kubo, due to a typo in the test, we miss a bug ipfs/kubo#9651
It's not crazy to imagine something similar could happen here.
The suite would be green, but the code wouldn't be tested.
We also need extra care when we port kubo tests: we can't do red/green testing.
So as a test writer, I have to introduce bugs (typos to values, wrong payloads, etc) to start from a red test and make sure it's working.
This PR proposes a "randomizer", which lives in front of a gateway and randomizes its outputs (headers and body).
If we run the test suite against our randomizer, we expect every test to fail most of the time.
If the entire test suite passes, there is a problem.
Next Steps