Skip to content

Commit

Permalink
test: ab-ify test_vulnerabilities.py
Browse files Browse the repository at this point in the history
These tests can fail due to external factors (microcode updates, AMI
updates, etc), which would then block our PR CI until those get
resolved. By using A/B-testing for our PR CI we avoid this, and get
alerted to these changes out-of-band.

Since A/B-Testing needs microvms compiled from different revisions, we
need to change our fixture approach a bit. Instead of building microvms,
it now provides factory methods that can be consumed by the A/B-test
functions for building microvms from compiled firecracker binaries.
These factory methods can then be composed to make them perform
additional actions such as "restore from snapshot" or "make sure checker
script is there".

The condition that the A/B-Tests verify is "PR did not introduce a
vulnerability". This is different from the "Result of vulnerability test
did not change across PR" that might be more obviously associated with
A/B-testing. However, this latter approach would not allow us to fix
vulnerabilities (as it would block such PRs).

Signed-off-by: Patrick Roy <[email protected]>
  • Loading branch information
roypat committed Oct 5, 2023
1 parent f12ff82 commit 6809cd6
Showing 1 changed file with 201 additions and 171 deletions.
Loading

0 comments on commit 6809cd6

Please sign in to comment.