Use balloon statistics in the test that checks balloon deflates on OOM #4150
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportAll modified lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #4150 +/- ##
=======================================
Coverage 83.10% 83.10%
=======================================
Files 225 225
Lines 28605 28605
=======================================
Hits 23772 23772
Misses 4833 4833
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
bchalios
added
the
Status: Awaiting review
bchalios
changed the title
bchalios
force-pushed
the
fix_test_balloon
branch
from
2f52798 to
2b70376
Compare
zulinx86
previously approved these changes
Oct 5, 2023
bchalios
force-pushed
the
fix_test_balloon
branch
5 times, most recently
from
9568a20 to
893e2e1
Compare
bchalios
force-pushed
the
fix_test_balloon
branch
from
893e2e1 to
48534eb
Compare
zulinx86
previously approved these changes
Oct 5, 2023
roypat
previously approved these changes
Oct 6, 2023
pb8o
previously approved these changes
Oct 6, 2023
Balloon devices have a feature where they can start deflating when the guest is in an OOM situation. We have a test that ensures this functionality works as expected. The test creates a microVM with a balloon device enabled, it inflates the balloon and then invokes a process in the microVM that exhausts the remaining microVM memory. The expectation is that the OOM killer will kick in and reap that process. The test relies on observing the process that fills up the memory to be killed in order to succeed. However, we do not really have control on what process the OOM will decide to kill, in low memory situations. This makes the test failing intermittently. This commit, changes the test to instead look into balloon statistics. Conceptually this makes sense; we don't want to test the OOM killer functionality, we want to ensure that the balloon device gives back memory to the VM in low memory situations. The balloon statistics can give us this information. Signed-off-by: Babis Chalios <[email protected]>
In test_balloon.py we are trying to find the PID of the SSH daemon process, so that we later change its OOM score, so that it does not get killed when we (deliberately) exhaust the available memory in a microVM. We use "pidof sshd" to do that. However, pidof returns the PIDs of all threads of the process. Now sshd launches a new thread for every new SSH connection. Later when we iterate over these PIDs to change their OOM score not all of the threads might be there, so the choom will fail for these. This is not really a problem, but it some times leads to misleading error messages. This commit drops the use of "pidof" in favour of reading the daemon's PID from "/run/sshd.pid". Signed-off-by: Babis Chalios <[email protected]>
In test_balloon.py::test_deflate_on_oom we are exhausting the memory of the microVM trying to trigger the OOM killer. This commit removes SSH commands after launching the memory hogger inside the microVM, to avoid hang connections due to the OOM killer killing sshd. Signed-off-by: Babis Chalios <[email protected]>
We have a mechanism that allows us to run a command inside a microVM. This mechanism ends up using Popen.communicate to retrieve the output of the SSH command. Popen.communicate comes with a timeout variable that we were not using. However, it is useful in cases where we don't want to wait for the result of the command we execute in the microVM. This commit extends our SSH mechanism to accept the timeout argument. Then, it uses the timeout in the test_balloon.py::test_deflate_on_oom when launching the fillmem process. fillmem drains the memory of the VM, which sometimes results in the SSH connection hanging. Signed-off-by: Babis Chalios <[email protected]>
bchalios
force-pushed
the
fix_test_balloon
branch
from
0095e0e to
57ed015
Compare
roypat
approved these changes
Oct 6, 2023
pb8o
approved these changes
Oct 6, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Reason
Balloon devices have a feature where they can start deflating when the guest is in an OOM situation. We have a test that ensures this functionality works as expected. The test creates a microVM with a balloon device enabled, it inflates the balloon and then invokes a process in the microVM that exhausts the remaining microVM memory. The expectation is that the OOM killer will kick in and reap that process. The test relies on observing the process that fills up the memory to be killed in order to succeed.
However, we do not really have control on what process the OOM will decide to kill, in low memory situations. This makes the test failing intermittently.
Changes
This PR, changes the test to instead look into balloon statistics. Conceptually this makes sense; we don't want to test the OOM killer functionality, we want to ensure that the balloon device gives back memory to the VM in low memory situations. The balloon statistics can give us this information.
By doing that, the test always passes when we configure the balloon device to deflate when the guest is in OOM conditions. However, the test is still flaky when we run it with the deflate on OOM option disabled. The reason is that some times the SSH command we run to spawn the guest process that drains memory hangs. Probably the OOM killer chooses the thread that handles the SSH connection. So, the PR also adds a timeout option in the function that we have to run commands over SSH inside the guest.
License Acceptance
By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache 2.0 license. For more information on following
Developer Certificate of Origin and signing off your commits, please check
CONTRIBUTING.md.PR Checklist
CHANGELOG.md.TODOs link to an issue.rust-vmm.