Experience trying AuthType auth/slurm over auth/munge

[dsloanm]

Of interest, Slurm 23.11 contains a new, built-in plugin for creating and validating credentials, i.e. AuthType = auth/slurm. This is positioned as an alternative to auth/munge, although "MUNGE is currently the default and recommended option" by the Slurm developers.

Part of this new authentication mechanism is the daemon sackd, intended for use on login nodes not running a full slurmd to enable both authentication via auth/slurm and retrieval of Slurm configuration files from slurmctld when running configless (as Charmed-HPC does).

As Slurm 23.11 is available in the Ubuntu 24.04 repos, including sackd, I set up a number of 24.04 containers in LXD with typical Slurm+MUNGE, switched over to auth/slurm, then created a new sackd-enabled login node. No real issues were found -- I was impressed at how smoothly the switchover went, at least in this toy example.

I've shared my experience/setup steps in the collapsible blocks below. MUNGE is still a dependency of the slurm-client, slurmctld, slurmd, etc. deb packages but I can see scope for that to change in future.

Cluster Setup

LXD containers:

• head-0 (10.212.98.197) running slurmctld
• compute-0 (10.212.98.102), compute-1 (10.212.98.65), compute-2 (10.212.98.189) compute nodes running slurmd
• ldap-0 (10.212.98.168) providing a directory service for consistent user identities across the cluster
• nfs-0 (10.212.98.109) providing a shared NFS file system

The full /etc/slurm/slurm.conf on head-0 is below:

Full slurm.conf

SlurmctldHost=head-0(10.212.98.197)
ClusterName=micro-hpc

AuthInfo=/var/run/munge/munge.socket.2
AuthType=auth/munge

MailProg=/usr/bin/mail.mailutils
PluginDir=/usr/lib/x86_64-linux-gnu/slurm-wlm
PlugStackConfig=/etc/slurm/plugstack.conf.d/plugstack.conf
RebootProgram="/usr/sbin/reboot --reboot"
SelectType=select/cons_tres
SlurmctldLogFile=/var/log/slurm/slurmctld.log
SlurmctldParameters=enable_configless
SlurmctldPidFile=/var/run/slurmctld.pid
SlurmctldPort=6817
SlurmdLogFile=/var/log/slurm/slurmd.log
SlurmdPidFile=/var/run/slurmd.pid
SlurmdPort=6818
SlurmdSpoolDir=/var/spool/slurmd
SlurmdUser=root
SlurmUser=slurm
StateSaveLocation=/var/spool/slurmctld

# Node Configurations
NodeName=compute-0 NodeAddr=10.212.98.102 CPUs=1 RealMemory=1000 TmpDisk=10000
NodeName=compute-1 NodeAddr=10.212.98.65 CPUs=1 RealMemory=1000 TmpDisk=10000
NodeName=compute-2 NodeAddr=10.212.98.189 CPUs=1 RealMemory=1000 TmpDisk=10000

# Partition Configurations
PartitionName=all Nodes=compute-0,compute-1,compute-2 MaxTime=30 MaxNodes=3 State=UP

Switching to auth/slurm

On the host, generate slurm.key

dd if=/dev/random of=slurm.key bs=1024 count=1

Push file to all nodes running slurmctld and slurmd and set ownership/permissions:

nodes=( head-0 compute-0 compute-1 compute-2 )
for i in ${nodes[@]}; do
  lxc file push slurm.key $i/etc/slurm/slurm.key;
  lxc exec $i -- chown slurm:slurm /etc/slurm/slurm.key;
  lxc exec $i -- chmod 600 /etc/slurm/slurm.key;
done

Log into slurmctld head node

lxc shell head-0

Modify slurm.conf

vi /etc/slurm/slurm.conf

replacing:

AuthInfo=/var/run/munge/munge.socket.2
AuthType=auth/munge

with:

AuthType=auth/slurm
CredType=cred/slurm

then restart slurmctld

systemctl restart slurmctld

Log out from head-0 to return to the host then restart slurmd on all compute nodes

logout
computes=( compute-0 compute-1 compute-2 )
for i in ${computes[@]}; do lxc exec $i -- systemctl restart slurmd; done

Stop munge.service and move munge.key to be certain new auth method is being used

for i in ${nodes[@]}; do
  lxc exec $i -- systemctl stop munge.service;
  lxc exec $i -- mv /etc/munge/munge.key /root/;
done

Log into head-0 as a normal user and submit a test job

ssh [email protected]
cat << 'EOF' > test.submit
#!/bin/bash
#SBATCH --job-name=test
#SBATCH --partition=all
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=1
#SBATCH --time=00:00:30
#SBATCH --error=test.%J.err
#SBATCH --output=test.%J.out
echo "Hello from `hostname` using auth/slurm!"
EOF
sbatch test.submit

Check the output

cat test.<job_id>.out

Example output:

Hello from compute-0 using auth/slurm!

A job was successfully submitted and run. Success!

Creating a login node

Copy an existing compute node to a new container login-0

lxc copy compute-0 login-0 --instance-only
lxc start login-0

Log in and replace slurmd with sackd

lxc shell login-0
systemctl disable --now slurmd
apt install sackd

Create a /etc/default/sackd file pointing to slurmctld host head-0(10.212.98.197).

cat << 'EOF' > /etc/default/sackd
# Additional options that are passed to the sackd daemon
SACKD_OPTIONS="--conf-server 10.212.98.197:6817"
EOF

Note: this file does not exist by default but is referenced by the packaged sackd.service file. Attempting to start sackd without creating /etc/default/sackd fails with sackd.service: Referenced but unset environment variable evaluates to an empty string: SACKD_OPTIONS.

Enable and start sackd

systemctl enable --now sackd

Check sackd has successfully pulled slurm.conf from head-0:

head -n 2 /run/slurm/conf/slurm.conf

Example output:

SlurmctldHost=head-0(10.212.98.197)
ClusterName=micro-hpc

Confirm Slurm commands work

sinfo

Example output:

   all          up      30:00      3   idle compute-[0-2]

Submit a test job as a normal user (test.submit created earlier on head-0 is available on login-0 through the shared NFS file system)

sudo -i -u normaluser
sbatch test.submit

Check the output

cat test.<new_job_id>.out

Example output:

Hello from compute-1 using auth/slurm!

Success again!

[jamesbeedy]

All about this! I'll give it a go and report back!

[NucciTheBoss]

Hmm... I'm interested to know why MUNGE is still the recommended default authentication plugin, even after the introduction of auth/slurm. I wonder if it's because there's just more documentation and user guides out there about how to set up Slurm with MUNGE while, as we found with sackd, there's not that much documentation out there yet?

In regards to info that needed to be exchanged between the login node (sackd) and the controller (slurmctld), is it just the following?

1. The slurmctld node hostname and port
2. The slurm.key secret

I could see us wanting to use sackd on the login nodes because (a) we can stop worrying about managing both MUNGE and Slurm and (b) we don't have to get creative with the back-end to ensure that the login node isn't enlisted as a compute node.

The required changes to hpc-libs is also pretty minimal as we just need to add an additional service type + SackdManager to slurm_ops. Big thing would be understanding how the current set of relations need to be reworked to support using auth/slurm instead of auth/munge.

[dsloanm]

My guess is MUNGE is more heavily tested just by virtue of having been around longer, so would be the recommended default until auth/slurm is more proven. E.g., looking through the Slurm bug tracker, there's been teething issues like incompatibility between auth/slurm and pam_slurm_adopt (fixed in 23.11.5).

Can confirm only the slurmctld hostname:port and slurm.key were needed to get the login node running. I've also found sackd appears to work alongside auth/munge, just as a manager for Slurm configuration files. I switched the test cluster back to MUNGE, set up both munge.service and sackd.service on the login node and submitted a job without issue. No need for a full slurmd on the login node or any back-end creativity (i.e. no login node NodeName entry needed in slurm.conf).

One comment is it was necessary to remove /etc/slurm/slurm.key on all nodes to switch back to MUNGE. Otherwise slurmd and sackd would fail to start with error: _fetch_child: failed to fetch remote configs: Unexpected message received, even with auth/slurm lines removed from slurm.conf and munge.service running. It seems if /etc/slurm/slurm.key exists, auth/slurm is prioritised and slurmd/sackd ignore /etc/munge/munge.key.

[NucciTheBoss]

Hmm... so if auth/munge works with sackd, I think it's entirely possible to create a charm for sackd then to function as our login node operator. Seems like the core functionality we need for an initial sackd operator is:

1. sackd + munge
2. sssd (provided by sssd-operator)
3. sshd (provided by the openssh-server package).

As for the upgrade story, if MUNGE is ever deprecated and/or auth/slurm becomes the preferred authentication system, it seems like the strategy is to:

1. Turn off the Slurm daemons.
2. Turn off MUNGE.
3. Remove the munge.key secret and remove MUNGE from all the nodes.
4. Update AuthType to auth/slurm.
5. Share slurm.key across the cluster.
6. Reactivate Slurm.

As for our dependencies to support sackd in Charmed HPC, it looks like we just need to do the following:

1. Upgrade the charms to support Noble so we can pull sackd from the main package archives.
2. Update hpc-libs to include a SackdManager.
3. Add a login-node relation to slurmctld-operator that exports the munge key as a Juju secret and hostname:port of the slurmctld node.

[dsloanm]

Yep, that all sounds good to me.

On another note, I've been trying the AuthInfo=use_client_ids option for auth/slurm. It allows for "LDAP-less" setups: I can shut down the ldap-0 container, stop sssd on all nodes and successfully run jobs as a local user created on login-0 with useradd. Jobs run on the compute nodes and create output files with the expected owner/group IDs, even though the user only exists on the login node. This opens the door to not strictly needing sssd or a directory service.

[NucciTheBoss]

I did see the client ids option. I think for now will march ahead with the SSSD implementation because it's much easier to model.

I'd be interested to see what an "SSSD-less" setup would look like - e.g. just sync /etc/passwd between nodes - but I think its a little too open-ended at the moment compared to LDAP + SSSD.

No matter how fast you run, you can't escape LDAP...

[dsloanm]

Something to be aware of, the sackd docs say it runs as SlurmdUser but seems it's actually SlurmUser:

root@login-0:~# grep User /run/slurm/conf/slurm.conf
SlurmdUser=root
SlurmUser=slurm
root@login-0:~# ps aux | grep sackd
slurm        158  0.0  0.2 154900  4096 ?        Ssl  10:19   0:00 /usr/sbin/sackd --systemd --conf-server 10.212.98.197:6817

Edit: nope! It's the sackd.service included with the deb setting User=slurm and Group=slurm.

[NucciTheBoss]

Makes sense. Currently we have root as the SlurmdUser, but it's a bad idea to have daemons that are "public facing" to be running as root. We'll need to see what the future ramifications are for running sackd as SlurmUser instead of SlurmdUser, but it seems fine at least right now 😅

[NucciTheBoss]

Through our game of telephone with SchedMD at SC24 (thanks Quinn), it looks like MUNGE working with sackd is a happy accident rather than an intended feature: https://mast.hpc.social/@quinn/113522246566164617

Not necessarily reassuring to hear that, but it seems that auth/munge with sackd is not something we should expect to continue working with each release of Slurm as it doesn't seem like something they test to ensure it works before release.

As for why MUNGE is still the recommended default, it's because SchedMD wants to provide a transition period for migrating from auth/munge to auth/slurm. I think it's safe to say that MUNGE will be deprecated in favor of auth/slurm relatively soon since it looks like they solved a lot of the teething issues. We have a choice to make here:

1. Continue using MUNGE with the expectation that we'll need to remove it in 1-2 years, and we'll need to provide support for security fixes + maintenance for MUNGE as auth/slurm will become the preferred authentication mechanism.
2. Commit to using auth/slurm now, and update the Slurm charm relations to use slurm_key (or just key) instead of munge_key.

Personally, I'm leaning towards option 2 as I'd rather help dogfood the new authentication mechanism, and we have to do less work later to support a migration path when MUNGE is deprecated for realz. Also, it's one less service that we have to manage on all of the Slurm nodes or applications that need to be able to talk to Slurm such as Open OnDemand.

Thoughts @jamesbeedy @dsloanm?

[dsloanm]

Yes, I'd also say option 2. For the reasons you've given and since sackd simplifies login node creation. Without sackd, our options are:

• slurmd on the login node, with a NodeName entry in slurm.conf but not added to any partition to prevent jobs being scheduled. Ref: https://support.schedmd.com/show_bug.cgi?id=9832
• No daemon on the login node and every client command pulls the config from the controller when run, which can result in excessive traffic to the controller. Ref: https://slurm.schedmd.com/configless_slurm.html

Both seem less appealing than adopting auth/slurm.

[NucciTheBoss]

• No daemon on the login node and every client command pulls the config from the controller when run, which can result in excessive traffic to the controller. Ref: https://slurm.schedmd.com/configless_slurm.html

Yes, let's definitely not do that 👀

• slurmd on the login node, with a NodeName entry in slurm.conf but not added to any partition to prevent jobs being scheduled. Ref: https://support.schedmd.com/show_bug.cgi?id=9832

The other downside to this approach is that we'll need to trigger a reload of all the configuration files with scontrol reconfigure for the slurmd daemons when new login nodes are added. Otherwise all the slurmd daemons will complain that their downloaded copy of slurm.conf is out of sync with the controller.

[jamesbeedy]

Through our game of telephone with SchedMD at SC24 (thanks Quinn), it looks like MUNGE working with sackd is a happy accident rather than an intended feature: https://mast.hpc.social/@quinn/113522246566164617

Not necessarily reassuring to hear that, but it seems that auth/munge with sackd is not something we should expect to continue working with each release of Slurm as it doesn't seem like something they test to ensure it works before release.

As for why MUNGE is still the recommended default, it's because SchedMD wants to provide a transition period for migrating from auth/munge to auth/slurm. I think it's safe to say that MUNGE will be deprecated in favor of auth/slurm relatively soon since it looks like they solved a lot of the teething issues. We have a choice to make here:

1. Continue using MUNGE with the expectation that we'll need to remove it in 1-2 years, and we'll need to provide support for security fixes + maintenance for MUNGE as auth/slurm will become the preferred authentication mechanism.

2. Commit to using auth/slurm now, and update the Slurm charm relations to use slurm_key (or just key) instead of munge_key.

Personally, I'm leaning towards option 2 as I'd rather help dogfood the new authentication mechanism, and we have to do less work later to support a migration path when MUNGE is deprecated for realz. Also, it's one less service that we have to manage on all of the Slurm nodes or applications that need to be able to talk to Slurm such as Open OnDemand.

Thoughts @jamesbeedy @dsloanm?

I vote # 2 also.

[NucciTheBoss]

Okay, judging from our discussion here, it looks like our best course of action to use sackd as Charmed HPC's login node daemon is to adopt auth/slurm in the Slurm charms. I've identified a couple of action items that we need to act one:

1. Update the Slurm charms to Noble. We get the added benefit of a newer base, and we get a version of Slurm that supports auth/slurm. (blocked on slurm_ops supporting Noble + apt charm library fixes for Noble)
2. Implement SackdManager in hpc-libs (already in progress).
3. Add sackd charm to slurm-charms + published to latest/edge (already in progress).
4. Refactor the Slurm charms to use auth/slurm instead of auth/munge.
5. Update Slurm charm integration tests to not test MUNGE.
6. Remove MUNGE support from hpc-libs.

Thanks for the great discussion everyone!