malcolm-test serves to run an instance of Malcolm and verify the results of system tests executed against it. It consists mostly of a control script, TOML files containing provisioning steps for virtual machine creation, and the test files themselves. See this issue in the Malcolm repository for the discussion leading up to its creation.
Using pip, to install the latest release from PyPI:
python3 -m pip install -U malcolm-test
Or to install directly from GitHub:
python3 -m pip install -U 'git+https://github.com/idaholab/Malcolm-Test'
This project makes use of:
- virter for the creation and execution of libvirt-based virtual machines running Malcolm
- pytest for the testing framework
pytest will be installed as a dependency by pip, but you will need to install and configure virter prior to running mmalcolm-test. See its GitHub page for instructions.
When installed via pip, this script may be executed as malcolm-test from the Linux command line.
usage: malcolm-test <flags> <extra arguments for pytest>
See README.md for usage details.
options:
-h, --help show this help message and exit
--verbose, -v Increase verbosity (e.g., -v, -vv, etc.)
--version [true|false]
Show script version and exit
Malcolm Git repo:
-g, --github-url <string>
Malcolm repository url (e.g., https://github.com/idaholab/Malcolm)
-b, --github-branch <string>
Malcolm repository branch (e.g., main)
Virtual machine specifications:
-c, --cpus <integer> Number of CPUs for virtual Malcolm instance
-m, --memory <integer>
System memory (GB) for virtual Malcolm instance
-d, --disk <integer> Disk size (GB) for virtual Malcolm instance
-i, --image <string> Malcolm virtual instance base image name (e.g., debian-12)
--image-user <string>
Malcolm virtual instance base image username (e.g., debian)
--vm-name-prefix <string>
Prefix for Malcolm VM name (e.g., malcolm)
--existing-vm <string>
Name of an existing virter VM to use rather than starting up a new one
--vm-provision-os [true|false]
Perform VM provisioning (OS-specific)
--vm-provision-malcolm [true|false]
Perform VM provisioning (Malcolm-specific)
--vm-provision-path <string>
Path containing subdirectories with TOML files for VM provisioning
--build-vm <string> The name for a new VM image to build and commit instead of running one
--build-vm-keep-layers [true|false]
Don't remove intermediate layers when building a new VM image
Malcolm runtime configuration:
--container-image-file <string>
Malcolm container images .tar.xz file for installation (instead of "docker pull")
-s, --start [true|false]
Start Malcolm once provisioning is complete (default true)
-r, --rm [true|false]
Remove virtual Malcolm instance after execution is complete
--stay-up [true|false]
Stay running until CTRL+C or SIGKILL is received
--sleep <integer> Seconds to sleep after init before starting Malcolm (default 30)
Testing configuration:
--test-path <string> Path containing test definitions
-p, --pcap-path <string>
Path containing PCAP files used by tests (UPLOAD_ARTIFACTS in tests should resolve relative to this path)
-t, --run-tests [true|false]
Run test suite once Malcolm is started
Here are some examples of use cases for malcolm-test. The output here is shown without verbose logging enabled; increasing the level of verbosity with -v, -vv, etc., can provide more information about the operations being performed.
To provision the Malcolm VM from scratch, run the tests, then discard the VM:
$ malcolm-test --rm --pcap-path /path/to/Malcolm-Test-PCAP
====================== test session starts ======================
platform linux -- Python 3.13.0, pytest-8.3.3, pluggy-1.5.0
rootdir: <...>
collected 4 items
<...>/site-packages/maltest/tests/test_malcolm_db_health.py . [ 25%]
<...>/site-packages/maltest/tests/test_malcolm_exists.py . [ 50%]
<...>/site-packages/maltest/tests/test_malcolm_pcap.py . [ 75%]
<...>/site-packages/maltest/tests/test_malcolm_response.py . [100%]
====================== 4 passed in 0.26s ======================Explanation of arguments:
--rm: discard the VM after running the tests--pcap-path /path/to/Malcolm-Test-PCAP: specify the path to the upload artifacts used by the tests
Depending on the use case, because a complete from-scratch provisioning of the Malcolm VM can take a long time (due to installing packages, pulling Malcolm container images, waiting for startup, etc.), building a Malcolm image that can then be reused for multiple test runs may be useful. The steps to do this might look like this:
To build and provision a new Malcolm VM image and tag it as malcolm-testing:
$ malcolm-test --build-vm malcolm-testingTo run the tests using this already-provisioned VM image, then shut it down:
$ virter image ls
Name Top Layer Created
malcolm-testing sha256:fd2320408c79045dca9aa914f50858f686a740bd053d481c7f33b670d0d3f4c9 3 minutes ago
$ malcolm-test \
--rm \
--image malcolm-testing \
--vm-provision-os false \
--vm-provision-malcolm false \
--pcap-path /path/to/Malcolm-Test-PCAP
====================== test session starts ======================
platform linux -- Python 3.13.0, pytest-8.3.3, pluggy-1.5.0
rootdir: <...>
collected 4 items
<...>/site-packages/maltest/tests/test_malcolm_db_health.py . [ 25%]
<...>/site-packages/maltest/tests/test_malcolm_exists.py . [ 50%]
<...>/site-packages/maltest/tests/test_malcolm_pcap.py . [ 75%]
<...>/site-packages/maltest/tests/test_malcolm_response.py . [100%]
====================== 4 passed in 0.62s ======================Explanation of arguments:
--rm: discard the VM after running the tests--image malcolm-testing: use themalcolm-testingimage already built instead of building one from scratch--vm-provision-os false: since the image is already provisioned, skip VM provisioning steps--vm-provision-malcolm false: since the image already configured Malcolm, skip Malcolm provisioning steps--pcap-path /path/to/Malcolm-Test-PCAP: specify the path to the upload artifacts used by the tests
During test development, or in other instances where you wish to run the tests over and over again without bringing Malcolm up and down, this could be accomplished as described below.
In one shell session start a Malcolm VM using the already-provisioned image, keeping Malcolm running until interrupted, without running the tests:
$ malcolm-test \
--stay-up \
--rm \
--image malcolm-testing \
--vm-provision-os false \
--vm-provision-malcolm false \
--run-tests falseExplanation of arguments:
--stay-up: keep Malcolm up until interrupted (with CTRL+C or SIGKILL)--rm: discard the VM after shutting down--image malcolm-testing: use themalcolm-testingimage already built instead of building one from scratch--vm-provision-os false: since the image is already provisioned, skip VM provisioning steps--vm-provision-malcolm false: since the image already configured Malcolm, skip Malcolm provisioning steps--run-tests false: don't run the tests with this instance ofmalcolm-test, just start Malcolm and wait
In another shell session, connecting to the existing running Malcolm instance for a test run:
$ virter vm ls
Name ID Access Network
malcolm-nearby-satyr 254 default
$ malcolm-test \
--rm false \
--start false \
--sleep 0 \
--existing-vm malcolm-nearby-satyr \
--vm-provision-os false \
--vm-provision-malcolm false \
--run-tests \
--pcap-path /path/to/Malcolm-Test-PCAP
====================== test session starts ======================
platform linux -- Python 3.13.0, pytest-8.3.3, pluggy-1.5.0
rootdir: <...>
collected 4 items
<...>/site-packages/maltest/tests/test_malcolm_db_health.py . [ 25%]
<...>/site-packages/maltest/tests/test_malcolm_exists.py . [ 50%]
<...>/site-packages/maltest/tests/test_malcolm_pcap.py . [ 75%]
<...>/site-packages/maltest/tests/test_malcolm_response.py . [100%]
====================== 4 passed in 0.25s ======================--rm false: don't destroy the VM after running the tests (the other instance ofmalcolm-testwill handle that)--start false: since Malcolm was already started by the other instance ofmalcolm-test, don't attempt to start it--sleep 0: since Malcolm was already started, there is no need to sleep before beginning test execution--existing-vm malcolm-nearby-satyr: specify the name of the existing running VM obtained byvirter vm ls--vm-provision-os false: since the image is already provisioned, skip VM provisioning steps--vm-provision-malcolm false: since the image already configured Malcolm, skip Malcolm provisioning steps--run-tests: run the test suite--pcap-path /path/to/Malcolm-Test-PCAP: specify the path to the upload artifacts used by the tests
Repeat the previous command as many times as needed as you adjust your tests. When finished, return to the first shell session and press CTRL+C to terminate and discard the Malcolm VM.
If the malcolm-test script exits uncleanly for some reason and leaves orphaned running VMs, they can be cleaned up like this:
$ virter vm ls
Name ID Access Network
malcolm-nearby-satyr 254 default
$ virter vm rm malcolm-nearby-satyrTo list and clean up any leftover Malcolm image tags:
$ virter image ls
Name Top Layer Created
rested-krill sha256:cd98ad4f552ce98f2a9ab0429b5fbb701483bc2980b590c5c91ebca2a8935f99 27 minutes ago
robust-condor sha256:e5306d90f825787b8142dcc94816902dfee4698fe9c00bc55aa4406eb5d830f5 26 minutes ago
up-quagga sha256:3889161a396f8f6ef41c0605323af07e2a9820cc980660563551a4488f0a7a3c 23 minutes ago
malcolm-testing sha256:fd2320408c79045dca9aa914f50858f686a740bd053d481c7f33b670d0d3f4c9 3 minutes ago
$ virter image rm rested-krill robust-condor up-quaggaPackage source highlights (under ./src/maltest):
- 🐍
maltest.py- A Python script for running Malcolm in a VM with virter - 🗁
virter/- A directory structure containing TOML files for provisioning the virter VMs in which Malcolm will run. Its subdirectories are arranged thusly:- 🗁
debian-12/- A directory matching the name of the virter image (supplied tomaltest.pywith the-i/--imageargument)- 🗁
init/- TOML files for the initial steps of provisioning the OS (before setting up and starting Malcolm) - 🗁
fini/- TOML files for the final stages of provisioning the OS (after shutting down Malcolm)
- 🗁
- 🗁
malcolm-init/- Distribution-agnostic provisioning TOML files for setting up Malcolm prior to starting it - 🗁
malcolm-fini/- Distribution-agnostic provisioning TOML files for tearing down Malcolm after tests are complete
- 🗁
- 🗁
tests/- A directory structure containing the test definitions, built using the pytest framework
malcolm-test uses the pytest framework. Please familiarize yourself with pytest as you begin developing new tests for the project.
New tests should be placed in the ./src/maltest/tests/ directory. Tests have access to the connection information for the running Malcolm instance through fixtures provided by ./src/maltest/tests/conftest.py.
See the following tests for examples of how to access and use these fixtures:
- test_malcolm_response.py - querying the Malcolm API using the Requests library
- test_malcolm_db_health.py - querying the data store directly using the OpenSearch or Elasticsearch client
Use UPLOAD_ARTIFACTS to specify a PCAP file required by your test. This example test would succeed if both foobar.pcap and barbaz.pcap were uploaded to Malcolm and their hashes stored in the global pcap_hash_map:
UPLOAD_ARTIFACTS = ['foobar.pcap', 'barbaz.pcap']
def test_malcolm_pcap_hash(
pcap_hash_map,
):
assert all([pcap_hash_map.get(x, None) for x in UPLOAD_ARTIFACTS])As PCAP files are uploaded to Malcolm by malcolm-test, they are hashed and stored in pcap_hash_map which maps the PCAP filename to its hash. The PCAP file is renamed to the hex-representation of the hash digest and the file extension (e.g., if the contents of foobar.pcap hashed to 52b92cdcec4af0e1, the file would be uploaded as 52b92cdcec4af0e1.pcap). This is done to ensure that conflicting PCAP filenames among different tests are resolved prior to processing. Since Malcolm automatically assigns tags to uploaded PCAP files, this hash should be used as a filter for the tags field for any network log-based queries for data related to that PCAP file. This way your tests can have reproducible outputs without being affected by PCAPs for other tests.