integration

Integration Tests

Introduction

The tests/integration directory contains an automated test harness for Slurm-Mail. It makes use of the docker images created by the Docker files in the docker-slurm sub directory.

Adding new Slurm version support

When a new version of Slurm is released the Docker files in the docker-slurm sub directory need to be updated to the new version and the images created and pushed to the ghcr.io/neilmunday/slurm-mail image repository. This is partially automated via the slurm_version_check and build_docker_slurm workflows.

Each day slurm_version_check checks for new versions of Slurm. If a new version is found the workflow updates all of the files in the docker-slurm directory with the new version string and creates a pull request.

When the pull request is merged the build_docker_slurm workflow automatically creates the new Slurm-Mail docker images so they are ready to use.

Finally, the ../.github/workflows/testing.yml workflow must be updated to include the new Slurm version.

Adding tests

Slurm tests should be added to tests.yml.

A test is defined like so:

  testX:
    # commands to run in batch script
    commands: |
      srun hostname  
    # test description
    description: one node test
    # is the job expected to fail?
    job_fail: false
    # sbatch options
    options:
      mail-type: ALL
      time: "01:00"
    # optionally specify any commands to run after job submission
    post_submit: |
      echo "I was run"
    # are errors expected in the slurm-send-mail log?
    send_errors: false
    # are errors expected in the slurm-spool-mail log?
    spool_errors: false
    # how many spool files are expected?
    spool_file_total: 2

Running tests

The run.sh script is used to run the test suite against a particular version of Slurm.

The script performs the following tasks:

• Builds the Slurm-Mail RPM for the target OS
• Creates a temporary docker compose configuration to run a two node cluster
• Builds the head node image to include the Slurm-Mail RPM
• Starts the containers
• Initiates the test suite from the head node
• Stops the containers and removes the images

Within the image that is created for the head node, the mail-server.py script is used to create a simple mail server to process e-mails generated by Slurm-Mail.

The run-tests.py together with tests.yml are copied inside the head node image.

Once the head node container is up and running the run.sh script executes run-tests.py which submits the jobs defined in tests.yml. As each job runs the run-tests.py script checks if each job was processed as expected by Slurm-Mail.

A summary of how many tests passed and failed is printed at the end of the tests.

Excute all tests against Slurm 22.05.6:

./run.sh -s 22.05.6

Execute a particular test:

./run.sh -s 22.05.6 -t test2

Enable verbose logging:

./run.sh -s 22.05.6 -v

Tip: As run.sh will build a Slurm-Mail RPM you can use the -r flag on subsequent invocations to skip building the RPM and thus save time.

Testing e-mails

If you want to see how the e-mails will look in an e-mail client you can use the provided demo.sh script to start-up Slurm-Mail containers together with a MailHog container.

Usage:

./demo.sh -s SLURM_VERSION -o OS_VERSION [-p]

If the Slurm-Mail RPM has already been built and exists in this directory you can use -p option to skip building the RPM.

./demo.sh -s 24.05.0 -o el9

Once the containers are up and running you can access the MailHog web GUI at http://localhost:8025

To submit jobs you can run the following command to launch an interactive bash shell:

docker exec -it slurm-mail-el9-24.05.0-head /usr/bin/bash -i

Reverting to /usr/bin/mail

If you want to see how e-mails from Slurm look like instead, perform the following steps:

1. Disable the cron job at /etc/cron.d/slurm-mail

2. Install sendmail:

dnf install -y sendmail

3. Add to the end of /etc/mail.rc:

set smtp=mailhog:1025

4. Set MailProg in /etc/slurm/slurm/conf to /usr/bin/mail.

5. Restart slurmctld:

supervisorctl stop slurmctld
supervisorctl start slurmctld