Modify the staging k6 load testing to allow for accurate tests with lower resources #5179
Labels
💻 aspect: code
Concerns the software code in the repository
🧰 goal: internal improvement
Improvement that benefits maintainers, not users
🟧 priority: high
Stalls work on the project or its dependents
🧱 stack: frontend
Related to the Nuxt frontend
🔒 staff only
Restricted to staff members
Problem
The current frontend staging k6 load tests require the the staging tasks to have the CPU of 2 vCPU to run. When running with 0.25 vCPU, the tests fail before any memory leak can be detected due to CPU load being over 100%.
Description
Based on the suggestion from @sarayourfriend , we should scale the staging load tests down until they reliably pass, rather than staging tasks up.
To simplify finding the right balance of VU and number of iteration, we should try using the constant-arrival-rate executor with timeUnit: 1s, rate: 2, duration: 5m and then manually ramp rate up by +1 until staging starts to fall over. The goal would be to find the highest rate where it consistently passes so you don't end up with flaky tests, but still have an aggressive enough load test that something like a memory leak would show up.
To test if this testing mode catches a memory leak, we can intentionally merge in the changes reverted in #4864.
Trying the same with something CPU bound would also be good, but I don't know of such a change to use for a test.
Alternatives
Keep using the higher CPU values for staging. This would cost more.
Additional context
The text was updated successfully, but these errors were encountered: