MPI.jl fails to find homebrew MPI in macos Sonoma runner

[miguelbiron]

Hi folks -- as the title says, we're getting the following error in our CI workflow

https://github.com/Julia-Tempering/Pigeons.jl/actions/runs/8849574516/job/24301884110#step:7:1236

The same happens for openmpi. Not sure if this is an issue on your end or upstream (julia's Libdl).

[simonbyrne]

Why not just use the default MPI?

[miguelbiron]

Because at some point we did just that, and we realized that many of our users were finding issues that were not being picked up by our CI tests.

[miguelbiron]

BTW we basically copied the workflow from you, so we expect that if you rerun your CI you should see the same issues (your last run was 3 days ago and used the old runners).

[giordano]

Is this because macos-latest now is aarch64-darwin?

[sloede]

Is this because macos-latest now is aarch64-darwin?

I don't know if that's the cause, but yes, they are arm now

[giordano]

Would setting macos-13 instead of macos-latest help?

[sloede]

That or macos-12 worked for us (on a similar issue, but unrelated to MPI.jl)

[miguelbiron]

Testing this right now Julia-Tempering/Pigeons.jl#230. Went with macos-12 because this was used in the last successful run

[giordano]

We probably still want to understand what's happening, but currently I personally don't have the time to do that, but at least using an older macos image should help with getting a working environment.

[miguelbiron]

Ok confirmed that using macos-12 still works. Per @giordano's comment I'll leave the issue open until a more permanent solution arises.

[simonbyrne]

It's failing here as well:
https://github.com/JuliaParallel/MPI.jl/actions/runs/8807615431/job/24305646282

I don't really have the time to look into it at the moment though.

[giordano]

Ah, I think I understand now: up to the macos-13 image, the brew commands used to install packages in /usr/local (which is probably a default search path for the dynamic loader), but in macos-14 they're now installed under /opt/homebrew. I don't know whether this is a change in Homebrew, or macOS, or whatever, but the net result is that libmpi isn't anymore in a default search path for the dynamic loader, and so we have to do something #831 to reliably make loading libmpi. An alternative could be to tell Homebrew to install the packages again in /usr/local, but I think setting DYLD_FALLBACK_LIBRARY_PATH is slightly more general: we don't have to change the installation directory.

Originally posted by @giordano in #829 (comment)

[miguelbiron]

Thanks for fixing this! I'll borrow your solution for our own MPI tests.

[giordano]

I think #832 is slightly nicer as it doesn't involve messing with environment variables (but that requires a change in MPIPreferences.jl).

[miguelbiron]

Ah thanks yes I just saw that. Will switch to that approach once it is merged