Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
test: ab-ify test_vulnerabilities.py
These tests can fail due to external factors (microcode updates, AMI updates, etc), which would then block our PR CI until those get resolved. By using A/B-testing for our PR CI we avoid this, and get alerted to these changes out-of-band. Since A/B-Testing needs microvms compiled from different revisions, we need to change our fixture approach a bit. Instead of building microvms, it now provides factory methods that can be consumed by the A/B-test functions for building microvms from compiled firecracker binaries. These factory methods can then be composed to make them perform additional actions such as "restore from snapshot" or "make sure checker script is there". The condition that the A/B-Tests verify is "PR did not introduce a vulnerability". This is different from the "Result of vulnerability test did not change across PR" that might be more obviously associated with A/B-testing. However, this latter approach would not allow us to fix vulnerabilities (as it would block such PRs). Signed-off-by: Patrick Roy <[email protected]>
- Loading branch information