test: use single SSH connection for lifetime of microvm #4953
+1,641
−1,302
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Update Rust to 1.83 version Signed-off-by: Egor Lazarchuk <[email protected]> Signed-off-by: Patrick Roy <[email protected]>
From: Egor Lazarchuk <[email protected]> With a new Rust version, new Clippy entered our repository. This commit aligns our codebase with guidance provided by new lints. Also remove a `deny(clippy::pedantic)`, because I'm not fixing those. Signed-off-by: Egor Lazarchuk <[email protected]> Co-authored-by: Patrick Roy <[email protected]> Signed-off-by: Patrick Roy <[email protected]>
This is needed for CI to install previous version of the toolchain used on the main branch. This change will need to be reverted after this PR is merged. Signed-off-by: Egor Lazarchuk <[email protected]> Signed-off-by: Patrick Roy <[email protected]>
Add the rust-src component and the $(uname -m)-unknown-linux-musl targets for the nightly toolchain, and install python3-seccomp and rustfilt. Since the python bindings for libseccomp are not published to pip, we have to install it into the global python installation via apt-get, and then copy into our venv. Signed-off-by: Patrick Roy <[email protected]>
Build static version of libseccomp with `musl-gcc`. This is needed for our musl builds as the version shipped in the ubuntu package is not compiled with `musl-gcc` and produces linker errors. Signed-off-by: Egor Lazarchuk <[email protected]> Signed-off-by: Patrick Roy <[email protected]>
From: Jonathan Woollett-Light <[email protected]> Adds `cargo-udeps` and the Rust `nightly` toolchain to support it, to dev container. Signed-off-by: Jonathan Woollett-Light <[email protected]> Co-authored-by: Patrick Roy <[email protected]> Signed-off-by: Patrick Roy <[email protected]>
These python dependencies are used for an upcoming PR about reusing SSH connection in our SSH tests. Signed-off-by: Patrick Roy <[email protected]>
accumulate all the dockerfile changes into a new devctr version Signed-off-by: Patrick Roy <[email protected]>
This is needed to work around a bug in `VmFd::create_device`, where undefined behavior caused a miscompilation on newer rust toolchains. Signed-off-by: Patrick Roy <[email protected]>
With the devctr's python dependencies update, we pulled in a new pylint version, which has a new `too-many-positional-arguments` lint. Fixing this would be a significant refactor, so just suppress it, as it seems low-value. Signed-off-by: Patrick Roy <[email protected]>
With newer rust toolchains, rust will SIGABRT if an already closed file descriptor is closed in the the `Drop` implementation for `File`. This also applies to creating `File`s with invalid file descriptors. Thus fix tests that accidentally double-close fds, and remove those that explicitly construct `File`s with invalid file descriptors (they are redundant now anyway, because Rust would abort if this ever happened). Signed-off-by: Patrick Roy <[email protected]>
It seemingly finally got intelligent enough to realize we don't need all these backslashes on the brackets. Signed-off-by: Patrick Roy <[email protected]>
The unittests here rely on the cfg(not(test)) antipattern for mocking, which new rust compiler interprets as the production version of the structs being dead-code when running unittests. Refactoring this to eliminate mocking is difficult, because fdt unit tests rely on the mocks removing all host-specific information from the cpu FDT nodes (particularly cache information, which in prod is read from the host sysfs, but in test mode is some dummy mock values). Signed-off-by: Patrick Roy <[email protected]>
with new mdformat we no longer need to wrap level 2 headings in escaped brackets, just normal brackets. Signed-off-by: Patrick Roy <[email protected]>
My python version does not accept the precense of double quotes inside a double-quoted f-string. Use single quotes instead. Signed-off-by: Patrick Roy <[email protected]>
roypat
requested review from
xmarcalx,
kalyazin,
pb8o and
Manciukic
as code owners
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #4953 +/- ##
==========================================
- Coverage 84.07% 83.98% -0.10%
==========================================
Files 251 251
Lines 28059 27889 -170
==========================================
- Hits 23592 23422 -170
Misses 4467 4467
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
roypat
added
the
Status: Awaiting review
zulinx86
previously approved these changes
Dec 11, 2024
tests/host_tools/network.py
Outdated
Comment on lines
79
to
74
|
|
||
| def _scp(self, path1, path2, options): | ||
| """Copy files to/from the VM using scp.""" | ||
| self._exec(["scp", *options, path1, path2], check=True) |
There was a problem hiding this comment.
nit: Do we still need _scp()? If no, we can also remove the internal function _exec() and hardcode the logic in run().
There was a problem hiding this comment.
Ah, yeah, good catch! Removed :)
Instead of creating new SSH connections every time we want to run a command inside the microvm, open a single ssh connection in the constructor of `SSHConnection`, and reuse it until we kill the microvm. Use the `fabric` SSH library to achieve this. Since fabric by default does not support opening its connections in specific network namespaces (e.g. ip netns exec), explicitly switch into the target network namespace before establishing the connected. Since entering network namspaces happens per-thread (instead of per-process), this is fine to do even in a highly-multithreaded pytest environment. Signed-off-by: Patrick Roy <[email protected]>
|
I'm closing this PR since @pb8o has shown me an approach using ControlMaster/ControlPersist that doesnt require new dependencies. Will open a separate PR with that. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
In the
SSHConnectionclass in our integration test framework, open a single SSH connection and reuse that for all communication with a specific microvm.Reason
Currently, we were recreating SSH connections for each command we want to run inside a microvm. This was leading to spurious failures because only the first connection attempt (which we also use to wait for microvm boot) was using retries.
License Acceptance
By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache 2.0 license. For more information on following Developer
Certificate of Origin and signing off your commits, please check
CONTRIBUTING.md.PR Checklist
tools/devtool checkstyleto verify that the PR passes theautomated style checks.
how they are solving the problem in a clear and encompassing way.
in the PR.
CHANGELOG.md.Runbook for Firecracker API changes.
integration tests.
TODO.rust-vmm.