Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add script to test S1 performance #24

Open
wants to merge 16 commits into
base: main
Choose a base branch
from
+113 −4
Open

Add script to test S1 performance #24

Changes from 1 commit
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
a93c208
Add script to test S1 performance
ocaisa Jun 7, 2023
65733d8
Refactor and improve
ocaisa Jun 8, 2023
2b94a66
Update run.sh
ocaisa Jun 8, 2023
ee00060
Update run.sh
ocaisa Jun 8, 2023
559dffa
Update run.sh
ocaisa Jun 8, 2023
811e45c
Update run.sh
ocaisa Jun 8, 2023
5b990cb
Update test_S1_performance.sh
ocaisa Jun 9, 2023
50771b4
Also save arch
ocaisa Jun 9, 2023
7a21f4e
Make script executable
ocaisa Jun 9, 2023
29286ed
Try to make S1 performance test restore previous setup, and run it in CI
ocaisa Jul 6, 2023
c38ad93
Be more careful about our location
ocaisa Jul 6, 2023
3ca2347
Only cleanup once!
ocaisa Jul 6, 2023
19840e5
Keep a copy of the json output in a file
ocaisa Jul 6, 2023
28d3c03
Upload a copy of the performance results as an artifact
ocaisa Jul 6, 2023
199801f
Tweak the output a tiny bit for readability
ocaisa Jul 6, 2023
e378f8b
Add number of files to json output
ocaisa Sep 28, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions scripts/test_S1_performance.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
#!/bin/bash
function join_by {
local d=${1-} f=${2-}
if shift 2; then
printf %s "$f" "${@/#/$d}"
fi
}
# Initialise EESSI
source /cvmfs/pilot.eessi-hpc.org/latest/init/bash > /dev/null
# Grab the date we do this and use that as out key for json output
date=$(date -I)
echo "{ \"$date\":["
json_array=()
for s1server in `grep CVMFS_SERVER_URL /etc/cvmfs/domain.d/eessi-hpc.org.conf | grep -o '".*"' | sed 's/"//g' | tr ';' '\n'`; do
# Edit the config file to point to a single S1 option, e.g.,
echo 'CVMFS_SERVER_URL="'"$s1server"'"' | sudo tee /etc/cvmfs/domain.d/eessi-hpc.org.local > /dev/null
# Reconfigure CVMFS
sudo cvmfs_config setup
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should have a way to check the user has the required sudo rights

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wasn't the idea to run this in a container, with bind mounts to empty directories for /var/lib/cvmfs & co (so empty cache), which would totally alleviate the need for sudo?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It wasn't my idea...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It actually doesn't seem like running it in a container can work:

Apptainer> cvmfs_config stat pilot.eessi-hpc.org    
pilot.eessi-hpc.org not mounted
Apptainer> source /cvmfs/pilot.eessi-hpc.org/latest/init/bash 
Found EESSI pilot repo @ /cvmfs/pilot.eessi-hpc.org/versions/2021.12!
archspec says aarch64/graviton2
Using aarch64/graviton2 as software subdirectory.
Using /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/aarch64/graviton2/modules/all as the directory to be added to MODULEPATH.
Found Lmod configuration file at /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/aarch64/graviton2/.lmod/lmodrc.lua
Initializing Lmod...
Prepending /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/aarch64/graviton2/modules/all to $MODULEPATH...
Environment set up to use EESSI pilot software stack, have fun!
[EESSI pilot 2021.12] $ cvmfs_config stat pilot.eessi-hpc.org
pilot.eessi-hpc.org not mounted
[EESSI pilot 2021.12] $ cvmfs_config wipecache
root privileges required
[EESSI pilot 2021.12] $ cvmfs_config setup    
root privileges required

I can get away with not wiping the cache since I control it, and I guess I can just overwrite the EESSI configuration file with a bind mount, but I am only left with timing the command if I can't run cvmfs_config stat

Copy link
Member Author

@ocaisa ocaisa Jun 9, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, more than that, I need to be able to run cvmfs_config setup to put the new configuration in place. Otherwise I need to bind mount a new configuration and run each S1 in a separate container execution step.

I really don't see the worth, we would lose a lot of valuable info and this implementation will run just fine as a CI job.

# Wipe the cache and run the example (I used Tensorflow from github.com/EESSI/eessi-demo)
sudo cvmfs_config wipecache >& /dev/null
# Run the TensorFlow example
cd ../TensorFlow
# Just print the real time
realtime=$({ ./run.sh > /dev/null ; } 2> >(grep real | awk '{print $2}'))
KB_per_sec=$(cvmfs_config stat pilot.eessi-hpc.org | column -t -H 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,17,18,19,20 | tail -1)
# Print json output
json_array+=("{\"$s1server\": [{\"time\":\"$realtime\"},{\"speed\":\"${KB_per_sec}KB/sec\"}]}")
Copy link
Member

@terjekv terjekv Jun 7, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For parsing reasons, could we do something like this instead?
json_array+=("{\"$s1server\": [{\"time\":\"$realtime\"},{\"speed\":\"${KB_per_sec}\",{\"speed_units\":\"KB/sec}\"}]}")

(I'm not sold on using JSON in command line environments though, I fear the parsing in general.)

done
join_by ,$'\n' "${json_array[@]}"
echo -e "\n]}"