Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add a randomizer proxy to test (some part of) the testsuite #29

Draft
wants to merge 12 commits into
base: main
Choose a base branch
from

Conversation

laurentsenta
Copy link
Contributor

@laurentsenta laurentsenta commented Mar 31, 2023

The worst thing that could happen to us is introducing a bug that disables one or more checks.

An example: in Kubo, due to a typo in the test, we miss a bug ipfs/kubo#9651

It's not crazy to imagine something similar could happen here.
The suite would be green, but the code wouldn't be tested.

We also need extra care when we port kubo tests: we can't do red/green testing.
So as a test writer, I have to introduce bugs (typos to values, wrong payloads, etc) to start from a red test and make sure it's working.

This PR proposes a "randomizer", which lives in front of a gateway and randomizes its outputs (headers and body).

If we run the test suite against our randomizer, we expect every test to fail most of the time.
If the entire test suite passes, there is a problem.

  • This is the first step towards "testing the test suite"; it requires little code & maintenance, but we can validate that we catch the most significant mistakes during local dev and in CI.
  • We implicitly get some benefits of red/green testing because we know that tests are actually testing things.
  • We could introduce more subtle bugs in payloads, content types, etc if we want to

Next Steps

  • validate the approach.
  • clean the code.
  • create an action that runs the test suite against the randomizer gateway and fails if the test suite passes.
  • create an action that raises warnings if a test passes.

@laurentsenta laurentsenta changed the base branch from main to feat/add-t0123 March 31, 2023 14:01
@laurentsenta laurentsenta marked this pull request as draft March 31, 2023 14:04
@laurentsenta laurentsenta changed the title feat: add a randomizer test feat: add a randomizer proxy to test (some part of) the tests Mar 31, 2023
@laurentsenta laurentsenta changed the title feat: add a randomizer proxy to test (some part of) the tests feat: add a randomizer proxy to test (some part of) the testsuite Mar 31, 2023
@laurentsenta
Copy link
Contributor Author

laurentsenta commented Mar 31, 2023

@lidel @galargh I'd like to get your feedback on that one,
it doesn't replace unit-tests but having a no-maintenance-catch-big-mistakes test would help me "sleep better" :)

@galargh
Copy link
Contributor

galargh commented Mar 31, 2023

I like it! That's really clever ❤️

@galargh
Copy link
Contributor

galargh commented Apr 1, 2023

Maybe we could run tests against random gateway a couple of times. Or rerun the tests that passed. Then, if we found a test case that passed on all the attempts, we'd know something's wrong.

Base automatically changed from feat/add-t0123 to main April 4, 2023 13:42
@laurentsenta laurentsenta mentioned this pull request Jul 3, 2023
29 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants