Issues with MPI: No Speedup on More Ranks

[hwhitehead]

I have successfully compiled Athena++ with the parallelised hdf5 library, and am attempting to run one of the test simulations (MBH blastwave, as described in the wiki tutorial) on a few courses to test the parallelisation.

However, I do not see any speedup when increasing the number of mpi tasks. In fact, the simulation takes longer to reach the same number of CPU cycles when run on more ranks. This is despite htop showing that the right number of tasks are being launched on runtime.

From my own brief testing, on 1, 2, 4 and 8 ranks it took the simulation ~1.5, 1.5, 2-3 and 7-8 minutes to reach ~300 CPU cycles, as tracked by the log file.

The simulation setup uses 8 MeshBlocks, each with 64^3 cells. Each time the run command has been:

mpiexec -np [numranks] ~/athena/bin/athena -i athinput.blast > log

I acknowledge that this may be an issue that leans more towards the hardware side, but I (and others) have run other simulation codes in parallel with mpi/hdf5 on this machine without issue. I could not find any mention of issues like this on the forum, is this an issue that other users have experienced?

It would also be useful to know if there are any other tools that might help breakdown exactly whats going on here. Is it possible that the code is just launching the same task multiple times?

Any advice on how to approach this issue would be much appreciated. I can provide file specifics if required, but I have been following the tutorial setup from this link exactly.

[c-white]

Does the log file show any signs of duplication, or is the output the same no matter how many processes are spawned?

Another thing to check is try turning off all outputs and see what the timing is then.

[hwhitehead]

Great catch, without any outputs turned on, the program runs properly with the runtime approximately halving as ranks double.

I presume this is an issue with my HDF5 setup in some way. I enquired with my machine administrator, the version of HDF5 installed is hdf5-1.12.0 from 2020 configured with --enable=parallel and --enable-fortran. Are there any known conflicts with particular versions?

As for the log files, there only appears to be one output stream each time. The only difference in the log file between runs is how rapidly the cycle steps are printed. When outputs are turned off, increasing ranks leads to faster printing. When outputs are turned on, the cycle steps tend to print in batches, pausing for longer when the number of ranks is increased.

[c-white]

There's no particular reason this should be slow, especially since these dumps are about 64 MB each.

I suggest gathering the following results and discussing them with your administrator: (1) timings without outputs, (2) timings with just hdf5, (3) timings with just vtk, and (4) timings with just restarts, where all the outputs are made with the same cadence. vtk outputs will be the same size as hdf5 in total, but each rank writes to a separate file with serial I/O commands. Restart dumps are about twice the size of hdf5 dumps; they are written to a single file but using MPI I/O commands rather than any extra library.

If the problem is with the hdf5 library, (2) will be the only slow case. If the problem is with your filesystem's ability to deal with explicitly parallel writes, (2) and (4) will be slow. If the problem is with your filesystem's speed overall, then (2), (3), and (4) will all be slow.

[hwhitehead]

Cheers, I'll gather this information and discuss it with my administrator. Leaving the thread open for now, but will attach a solution if/when I locate the issue.

Thanks again!

[hwhitehead]

The issue was as suggested, the filesystem I was attempting to save to was ill equipped for parallel writing. Moving the output destination to a different area with better I/O capability resulted in the usual MPI speedup when writing HDF5 files. Marking as solved now. Cheers!