[CI] oneAPI tests are hanging

[giordano]

CI jobs using Intel oneAPI MPI 2021.7.0 are hanging forever after the test_shared_win.jl test set, causing jobs to eventually time out

• last fully successful CI run, on 2023-11-06: https://github.com/JuliaParallel/MPI.jl/actions/runs/6767002150/job/18391882937 (commit: b3b83cd). This was using julia v1.9.3.
• first CI run where jobs started to hang, on 2023-11-29: https://github.com/JuliaParallel/MPI.jl/actions/runs/7038138525/job/19154435088 (after commits 3f26709 and c24db42, which only touched documentation, or docstrings and comments, no actual code). This was using julia v1.9.4.

To summarise

• between the two different runs the last julia versions changed, but on 2024-02-02 I ran again jobs with julia v1.9.3 in https://github.com/JuliaParallel/MPI.jl/actions/runs/7759323957 and the tests are hanging again, so the julia version doesn't seem to be relevant
• there haven't been significant changes of source code in this package between the two runs, so I would rule out some problems in this package
• more interestingly, I can't reproduce the issue locally in the containers we're using for CI (tried both ghcr.io/juliaparallel/github-actions-buildcache:intel-oneapi-mpi-jq or ghcr.io/juliaparallel/github-actions-buildcache:intel-oneapi-mpi-2021.7.0-gzc7es2p27ftwyk4sdplynlj6d54xzi6.spack): I installed both julia v1.9.4 and v1.9.3, I couldn't reproduce the hangs with either of them, tests just run fine every single time.

I suspect there have been some changes in GitHub-hosted runners configuration which is causing this, but this hypothesis is hard to test. One last option to try is to use a newer version of Intel oneAPI MPI, in case this was some sort of old bug later fixed, but I'll postpone that test for another day. In the meantime, I'm opening this ticket to keep track of the issue. Edit: it does appear that just upgrading to oneAPI 2021.11.0 magically fixes the hang without any other change on our side, this is implemented in #818.

[giordano]

With #818 merged we're back to green CI on master after three months: https://github.com/JuliaParallel/MPI.jl/actions/workflows/UnitTests.yml?query=branch%3Amaster 🙂