Running on Lassen

Lassen/LSF

Unlike Quartz, Lassen uses the LSF batch system. See docs at LLNL for more information.

Here is a basic working example for running a GPU build of the tps code using 4 GPUs (and 4 MPI tasks) per node:

$ cat job.lsf

#BSUB -nnodes 1                  #number of nodes
#BSUB -W 10                      #walltime in minutes
#BSUB -o job.%J.out
#BSUB -J tps
#BSUB -q pbatch

#-----------------------
# Setup a clean tps env
#-----------------------
module purge
module swap xl gcc/8.3.1
export SW_HOME=/usr/workspace/utaustin/software/lassen
export MODULEPATH=$SW_HOME/modules/gcc:$MODULEPATH
module load grvy mfem-cuda hypre metis cuda/11.2.0

#-----------------------------------------
# run the beast (4 mpi tasks [gpus]/node)
#-----------------------------------------

lrun -T 4 --gpubind=off ./tps.gpu --runFile runfile -d cuda

To submit the above job script, issue:

bsub < job.lsf

In general, the thing to change in the example script is the number of compute nodes desired (the first line of the job script). The total number GPUs and MPI ranks will be 4X this node count.