Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add script to test S1 performance #24

Open
wants to merge 16 commits into
base: main
Choose a base branch
from
Open

Add script to test S1 performance #24

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
a93c208
Add script to test S1 performance
ocaisa Jun 7, 2023
65733d8
Refactor and improve
ocaisa Jun 8, 2023
2b94a66
Update run.sh
ocaisa Jun 8, 2023
ee00060
Update run.sh
ocaisa Jun 8, 2023
559dffa
Update run.sh
ocaisa Jun 8, 2023
811e45c
Update run.sh
ocaisa Jun 8, 2023
5b990cb
Update test_S1_performance.sh
ocaisa Jun 9, 2023
50771b4
Also save arch
ocaisa Jun 9, 2023
7a21f4e
Make script executable
ocaisa Jun 9, 2023
29286ed
Try to make S1 performance test restore previous setup, and run it in CI
ocaisa Jul 6, 2023
c38ad93
Be more careful about our location
ocaisa Jul 6, 2023
3ca2347
Only cleanup once!
ocaisa Jul 6, 2023
19840e5
Keep a copy of the json output in a file
ocaisa Jul 6, 2023
28d3c03
Upload a copy of the performance results as an artifact
ocaisa Jul 6, 2023
199801f
Tweak the output a tiny bit for readability
ocaisa Jul 6, 2023
e378f8b
Add number of files to json output
ocaisa Sep 28, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Bioconductor/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,4 @@

module load R-bundle-Bioconductor/3.11-foss-2020a-R-4.0.0

time Rscript dna.R
time -p Rscript dna.R
2 changes: 1 addition & 1 deletion GROMACS/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,4 +12,4 @@ fi
rm -f ener.edr logfile.log

# note: downscaled to just 1k steps (full run is 10k steps)
time gmx mdrun -s ion_channel.tpr -maxh 0.50 -resethway -noconfout -nsteps 1000 -g logfile
time -p gmx mdrun -s ion_channel.tpr -maxh 0.50 -resethway -noconfout -nsteps 1000 -g logfile
2 changes: 1 addition & 1 deletion OpenFOAM/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ foamDictionary -entry runTimeModifiable -set "false" system/controlDict
foamDictionary -entry functions -set "{}" system/controlDict

mpirun --oversubscribe -np $NP potentialFoam -parallel 2>&1 | tee log.potentialFoam
time mpirun --oversubscribe -np $NP simpleFoam -parallel 2>&1 | tee log.simpleFoam
time -p mpirun --oversubscribe -np $NP simpleFoam -parallel 2>&1 | tee log.simpleFoam

echo "cleanup..."
rm -rf $WORKDIR
2 changes: 1 addition & 1 deletion TensorFlow/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,4 @@

module load TensorFlow/2.3.1-foss-2020a-Python-3.8.2

time python TensorFlow-2.x_mnist-test.py
time -p python TensorFlow-2.x_mnist-test.py
38 changes: 38 additions & 0 deletions scripts/test_S1_performance.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
#!/bin/bash

function join_by {
local d=${1-} f=${2-}
if shift 2; then
printf %s "$f" "${@/#/$d}"
fi
}

function test_S1 {
# Edit the config file to point to a single S1 option, e.g.,
echo 'CVMFS_SERVER_URL="'"$1"'"' | sudo tee /etc/cvmfs/domain.d/eessi-hpc.org.local > /dev/null
ocaisa marked this conversation as resolved.
Show resolved Hide resolved
# Reconfigure CVMFS
sudo cvmfs_config setup
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should have a way to check the user has the required sudo rights

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wasn't the idea to run this in a container, with bind mounts to empty directories for /var/lib/cvmfs & co (so empty cache), which would totally alleviate the need for sudo?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It wasn't my idea...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It actually doesn't seem like running it in a container can work:

Apptainer> cvmfs_config stat pilot.eessi-hpc.org    
pilot.eessi-hpc.org not mounted
Apptainer> source /cvmfs/pilot.eessi-hpc.org/latest/init/bash 
Found EESSI pilot repo @ /cvmfs/pilot.eessi-hpc.org/versions/2021.12!
archspec says aarch64/graviton2
Using aarch64/graviton2 as software subdirectory.
Using /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/aarch64/graviton2/modules/all as the directory to be added to MODULEPATH.
Found Lmod configuration file at /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/aarch64/graviton2/.lmod/lmodrc.lua
Initializing Lmod...
Prepending /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/aarch64/graviton2/modules/all to $MODULEPATH...
Environment set up to use EESSI pilot software stack, have fun!
[EESSI pilot 2021.12] $ cvmfs_config stat pilot.eessi-hpc.org
pilot.eessi-hpc.org not mounted
[EESSI pilot 2021.12] $ cvmfs_config wipecache
root privileges required
[EESSI pilot 2021.12] $ cvmfs_config setup    
root privileges required

I can get away with not wiping the cache since I control it, and I guess I can just overwrite the EESSI configuration file with a bind mount, but I am only left with timing the command if I can't run cvmfs_config stat

Copy link
Member Author

@ocaisa ocaisa Jun 9, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, more than that, I need to be able to run cvmfs_config setup to put the new configuration in place. Otherwise I need to bind mount a new configuration and run each S1 in a separate container execution step.

I really don't see the worth, we would lose a lot of valuable info and this implementation will run just fine as a CI job.

# Wipe the cache and run the example (from github.com/EESSI/eessi-demo)
sudo cvmfs_config wipecache >& /dev/null
# Run the example
cd ../$2
# Just print the real time
realtime=$({ ./run.sh > /dev/null ; } 2> >(grep real | awk '{print $2}'))
bandwidth=( $(cvmfs_config stat pilot.eessi-hpc.org | column -t -H 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,17,18,19,20 ) )
cache_usage=( $( cvmfs_config stat pilot.eessi-hpc.org | column -t -H 1,2,3,4,5,6,7,9,10,11,12,13,14,15,16,17,18,19,20 ))
# Print json output
echo -n "{\"$1\": {\"time\":\"$realtime\",\"speed\":\"${bandwidth[1]}\",\"speed_unit\":\"${bandwidth[0]}\",\"data\":\"${cache_usage[1]}\",\"data_unit\":\"${cache_usage[0]}\",\"application\":\"$2\",\"arch\":\"$EESSI_SOFTWARE_SUBDIR\" }}"
}

# Initialise EESSI
source /cvmfs/pilot.eessi-hpc.org/latest/init/bash > /dev/null
# Grab the date we do this and use that as out key for json output
application="${APPLICATION:-TensorFlow}"
date=$(date -I)
json_array=()
for s1server in `grep CVMFS_SERVER_URL /etc/cvmfs/domain.d/eessi-hpc.org.conf | grep -o '".*"' | sed 's/"//g' | tr ';' '\n'`; do
json_array+=("$(test_S1 "$s1server" "$application")")
done
echo -e "{\"$date\":["
join_by ,$'\n' "${json_array[@]}"
echo -e "\n]}"