cgdm hearhear

• Overview
• Installing spack
• Installing software with spack
  • General notes
  • Installing lmod
  • Setting up module-related defaults for spack
  • Installing gcc and gfortran
  • Installing openmpi and mpich
  • (Re)creating the module files
  • Making a module file for clang-gfortran
  • Installing NetCDF
  • Installing some python things
  • Installing other convenience software
    • emacs
• Installing conda
• Known issues
  • Issues with duplicate rpaths

Overview

cgdm-hearhear is an M2-based Mac laptop that the ESMF team set up November, 2024. It is set up with spack and modules to facilitate testing multiple compilers and MPI libraries.

This wiki page documents how this machine was set up. Note that the spack installation is currently under Bill Sacks's home directory (/Users/sacks/spack), but this could be redone in a shared space if desired.

The steps documented here are skipping some instances of undoing / redoing things as I was trying to get things working. The steps I'm leaving out are ones that I expect did not impact the end result in any significant way.

The starting point for this was essentially a fresh machine: the Apple developer tools were installed, but there was nothing installed with either Homebrew or MacPorts.

Installing spack

I installed spack in /Users/sacks/spack via:

git clone -c feature.manyFiles=true --depth=2 https://github.com/spack/spack.git
cd spack
git fetch --tags
git checkout develop-2024-11-10

I then added the following to my .bash_profile:

. ${HOME}/spack/share/spack/setup-env.sh

Installing software with spack

General notes

I'm generally building the software stack with clang, only building things with gcc where needed, since I ran into trouble trying to build some lower-level libraries with gcc.

Installing lmod

The latest (8.7.37) seems to have a bug: the init files are empty, leading to a situation where the "module" command isn't available; so using the older 8.7.24.

spack install [email protected]

Put the following in .bash_profile after the spack setup line:

. $(spack location -i lmod)/lmod/lmod/init/bash

And this, to use the version of modules for hierarchical modules:

module use ${SPACK_ROOT}/share/spack/lmod/darwin-sonoma-aarch64/Core/

(This directory may not exist yet, but will soon.)

And also creating a space for my own personal modules:

module use ${HOME}/modules

So my full .bash_profile now looks like this:

# Spack stuff
. ${HOME}/spack/share/spack/setup-env.sh
. $(spack location -i lmod)/lmod/lmod/init/bash
module use ${SPACK_ROOT}/share/spack/lmod/darwin-sonoma-aarch64/Core/
module use ${HOME}/modules

Setting up module-related defaults for spack

I created the following file in ~/.spack/modules.yaml:

modules:
  default:
    enable: [lmod]
    lmod:
      hash_length: 0
      exclude_implicits: true
      exclude:
      - lmod
      include:
      - cmake
      - python
      - py-pip
      - py-pyyaml
      core_compilers:
      - apple-clang@=16.0.0
      core_specs:
      - cmake
      - python
      - py-pip
      - py-pyyaml
      hierarchy:
      - mpi
      projections:
        netcdf-c^mpi:       '{name}/{version}'
        netcdf-c:           '{name}-nompi/{version}'
        netcdf-fortran^mpi: '{name}/{version}'
        netcdf-fortran:     '{name}-nompi/{version}'

Notes on this:

• enable: [lmod] enables the creation of module files for packages that spack installs. (The spack documentation suggests running spack config add modules:default:enable:[lmod], which creates this setting.)
• hash_length: 0 removes the hash from the module names (see https://spack-tutorial.readthedocs.io/en/latest/tutorial_modules.html#change-module-file-naming); with the other changes here for hierarchical modules, this hash is unnecessary.
• exclude_implicits: true avoids creating modules for packages that are installed as dependencies, instead only creating modules for packages that are explicitly installed with spack install. This solves three issues:
  1. It avoids listing a bunch of lower-level, uninteresting modules when running module avail. (This can also be achieved with hide_implicits: true, but using hide_implicits instead of exclude_implicits leaves the following issues unresolved.)
  2. It avoids listing the lower-level, uninteresting modules when running module list after doing a module load of a module that depends on some of these lower-level modules.
  3. Perhaps most importantly, it avoids issues I was running into with auto-loading lower-level modules that were created with a different compiler -- specifically, loading the module of a library that was created with gcc/gfortran but which depended on libraries created with clang/gfortran. (I think there may have been inconsistencies between the module paths to these lower-level libraries in the module files for the dependent libraries, and the paths where these lower-level module files actually resided. It's possible that there would be a way to resolve this issue, but simply avoiding creating modules at all for these lower-level libraries avoided the problem.)
• exclude: lmod avoids creating a module file for lmod, since that seems unnecessary and kind of circular.
• The include list will come into play later, after we install more things: This lists some implicitly-installed packages for which we do want modules.
• core_compilers: apple-clang@=16.0.0 specifies that anything built with this core (i.e., system-level) compiler should appear in the 'Core' part of the module hierarchy and thus always available, rather than being dependent on loading any particular compiler module.
• The core_specs list, like the include list, will come into play later, after we install more things: This lists some packages that we want to appear in the compiler-independent Core modules despite having been built with a non-core compiler.
• hierarchy: mpi says to include the mpi library in the module hierarchy. So modules for libraries that depend on a given mpi library will only appear once you load that mpi library.
• The projections section with its netcdf specifications will become relevant once we install NetCDF in a future step, below. The important aspect of this is to specify a naming convention that distinguishes between mpi and no-mpi versions of netcdf. Without this, after loading a compiler and mpi library, there would be two versions of the netcdf modules available with the same name: one built with the mpi library and one built without mpi. The settings here use the default naming convention ({name}/{version}) for the netcdf libraries built with dependence on an mpi library, but add -nompi to the name for netcdf libraries built without this dependence on an mpi library.

Installing gcc and gfortran

The latest gcc has problems building on MacOS (https://github.com/spack/spack/issues/45628), so using version 14.1.0:

spack install [email protected]
spack load gcc
spack compiler find

I then needed to fix the compiler spec for spack's clang-gfortran combination: In ~/.spack/darwin/compilers.yaml, there are two blocks with spec: apple-clang@=16.0.0: one has f77 and fc set to null; the second has them set to the gcc-14.1.0 gfortran. I'm changing the spec name for the second to be apple-clang@=16.0.0-gfortran14.1.0. (See also https://spack-tutorial.readthedocs.io/en/latest/tutorial_configuration.html#compiler-configuration).

Installing openmpi and mpich

First installing these mpi libraries with clang-gfortran:

spack install openmpi %apple-clang@=16.0.0-gfortran14.1.0
spack install mpich %apple-clang@=16.0.0-gfortran14.1.0

And then with gcc; note that these installations reused the dependencies installed using clang and just built the mpi libraries themselves using gcc. (This is important because some of the dependencies have build issues using gcc.):

spack install openmpi %[email protected]
spack install mpich %[email protected]

(Re)creating the module files

At this point and a few others, I had spack recreate its module files:

spack module lmod refresh --delete-tree -y

Making a module file for clang-gfortran

Since the clang-gfortran compiler combination wasn't something that spack installed, spack doesn't create a module file for it. We need a module file for two purposes:

1. Loading the necessary gcc/gfortran stuff when using clang-gfortran. (An alternative would be to load the gcc module, since we need to do most of the same stuff for both, and then set/unset environment variable(s) like CC. But that wouldn't accomplish the second purpose.)
2. Getting the clang-gfortran modules (like openmpi and mpich) to appear when we're using clang-gfortran, given the hierarchical module setup we're using.

I created a module for this, which I'm putting in my home directory, not under spack, so that when I ask spack to rebuild modules, it doesn't mess with this one. So creating file ${HOME}/modules/apple-clang/16.0.0-gfortran14.1.0.lua. This started as a copy of gcc/14.1.0.lua, but then I made some changes:

--- 14.1.0.lua	2024-11-15 14:05:52
+++ 16.0.0-gfortran14.1.0.lua	2024-11-15 14:05:48
@@ -1,27 +1,26 @@
 -- -*- lua -*-
--- Module file created by spack (https://github.com/spack/spack) on 2024-11-15 13:48:15.463589
+-- Module file created by modifying the gcc file created by spack (https://github.com/spack/spack) on 2024-11-14 11:44:29.184355
 --
 -- [email protected]%[email protected]~binutils~bootstrap~graphite~mold~nvptx~piclibs~strip build_system=autotools build_type=RelWithDebInfo languages='c,c++,fortran' patches=1529cff arch=darwin-sonoma-m2/yxln5vj
 --
 
-whatis([[Name : gcc]])
-whatis([[Version : 14.1.0]])
+whatis([[Name : apple-clang]])
+whatis([[Version : 16.0.0-gfortran14.1.0]])
 whatis([[Target : m2]])
-whatis([[Short description : The GNU Compiler Collection includes front ends for C, C++, Objective-C, Fortran, Ada, and Go, as well as libraries for these languages.]])
+whatis([[Short description : Use built-in clang for C/C++, and gfortran for Fortran.]])
 whatis([[Configure options : --with-pkgversion=Spack GCC --with-bugurl=https://github.com/spack/spack/issues --disable-multilib --enable-languages=c,c++,fortran --disable-nls --disable-canonical-system-headers --with-system-zlib --with-zstd-include=/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/zstd-1.5.6-qgzwdavw3b3q35p2jt7qu4yb5kmkbskr/include --with-zstd-lib=/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/zstd-1.5.6-qgzwdavw3b3q35p2jt7qu4yb5kmkbskr/lib --with-ld=/Library/Developer/CommandLineTools/usr/bin/ld-classic --disable-bootstrap --with-mpfr-include=/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/mpfr-4.2.1-k6l2mohdprx77kyqgknyfgyyv7tfi5nh/include --with-mpfr-lib=/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/mpfr-4.2.1-k6l2mohdprx77kyqgknyfgyyv7tfi5nh/lib --with-gmp-include=/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/gmp-6.3.0-xt3cjmh7dehpxzqwjklftlml6vdqica2/include --with-gmp-lib=/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/gmp-6.3.0-xt3cjmh7dehpxzqwjklftlml6vdqica2/lib --with-mpc-include=/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/mpc-1.3.1-r4htrqxnrh4ia7hj5na3uxxihujzcgqf/include --with-mpc-lib=/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/mpc-1.3.1-r4htrqxnrh4ia7hj5na3uxxihujzcgqf/lib --without-isl --with-native-system-header-dir=/usr/include --with-sysroot=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk --with-libiconv-prefix=/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/libiconv-1.17-6735mlktagraawr36ttz4rzm6i5p6lis --with-stage1-ldflags=-Wl,-rpath,/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/gcc-14.1.0-yxln5vjswuutfxtzpq4tf5yk7iuhin4a/lib -Wl,-rpath,/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/gcc-14.1.0-yxln5vjswuutfxtzpq4tf5yk7iuhin4a/lib64 -Wl,-rpath,/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/gmp-6.3.0-xt3cjmh7dehpxzqwjklftlml6vdqica2/lib -Wl,-rpath,/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/libiconv-1.17-6735mlktagraawr36ttz4rzm6i5p6lis/lib -Wl,-rpath,/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/mpc-1.3.1-r4htrqxnrh4ia7hj5na3uxxihujzcgqf/lib -Wl,-rpath,/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/mpfr-4.2.1-k6l2mohdprx77kyqgknyfgyyv7tfi5nh/lib -Wl,-rpath,/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/zlib-ng-2.2.1-n5joj3sdmckf3f545vscegafl34xo6rz/lib -Wl,-rpath,/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/zstd-1.5.6-qgzwdavw3b3q35p2jt7qu4yb5kmkbskr/lib --with-boot-ldflags=-Wl,-rpath,/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/gcc-14.1.0-yxln5vjswuutfxtzpq4tf5yk7iuhin4a/lib -Wl,-rpath,/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/gcc-14.1.0-yxln5vjswuutfxtzpq4tf5yk7iuhin4a/lib64 -Wl,-rpath,/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/gmp-6.3.0-xt3cjmh7dehpxzqwjklftlml6vdqica2/lib -Wl,-rpath,/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/libiconv-1.17-6735mlktagraawr36ttz4rzm6i5p6lis/lib -Wl,-rpath,/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/mpc-1.3.1-r4htrqxnrh4ia7hj5na3uxxihujzcgqf/lib -Wl,-rpath,/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/mpfr-4.2.1-k6l2mohdprx77kyqgknyfgyyv7tfi5nh/lib -Wl,-rpath,/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/zlib-ng-2.2.1-n5joj3sdmckf3f545vscegafl34xo6rz/lib -Wl,-rpath,/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/zstd-1.5.6-qgzwdavw3b3q35p2jt7qu4yb5kmkbskr/lib -static-libstdc++ -static-libgcc --with-build-config=spack]])
 
-help([[Name   : gcc]])
-help([[Version: 14.1.0]])
+help([[Name   : apple-clang]])
+help([[Version: 16.0.0-gfortran14.1.0]])
 help([[Target : m2]])
 help()
-help([[The GNU Compiler Collection includes front ends for C, C++, Objective-C,
-Fortran, Ada, and Go, as well as libraries for these languages.]])
+help([[Use built-in clang for C/C++, and gfortran for Fortran.]])
 
 -- Services provided by the package
 family("compiler")
 
 -- Loading this module unlocks the path below unconditionally
-prepend_path("MODULEPATH", "/Users/sacks/spack/share/spack/lmod/darwin-sonoma-aarch64/gcc/14.1.0")
+prepend_path("MODULEPATH", "/Users/sacks/spack/share/spack/lmod/darwin-sonoma-aarch64/apple-clang/16.0.0-gfortran14.1.0")
 
 
 
@@ -29,7 +28,7 @@
 prepend_path("PATH", "/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/gcc-14.1.0-yxln5vjswuutfxtzpq4tf5yk7iuhin4a/bin", ":")
 prepend_path("MANPATH", "/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/gcc-14.1.0-yxln5vjswuutfxtzpq4tf5yk7iuhin4a/share/man", ":")
 prepend_path("CMAKE_PREFIX_PATH", "/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/gcc-14.1.0-yxln5vjswuutfxtzpq4tf5yk7iuhin4a/.", ":")
-setenv("CC", "/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/gcc-14.1.0-yxln5vjswuutfxtzpq4tf5yk7iuhin4a/bin/gcc")
+setenv("CC", "/usr/bin/clang")
 setenv("FC", "/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/gcc-14.1.0-yxln5vjswuutfxtzpq4tf5yk7iuhin4a/bin/gfortran")
 setenv("F77", "/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/gcc-14.1.0-yxln5vjswuutfxtzpq4tf5yk7iuhin4a/bin/gfortran")
 append_path("MANPATH", "", ":")

Resulting in this file:

-- -*- lua -*-
-- Module file created by modifying the gcc file created by spack (https://github.com/spack/spack) on 2024-11-14 11:44:29.184355
--
-- [email protected]%[email protected]~binutils~bootstrap~graphite~mold~nvptx~piclibs~strip build_system=autotools build_type=RelWithDebInfo languages='c,c++,fortran' patches=1529cff arch=darwin-sonoma-m2/yxln5vj
--

whatis([[Name : apple-clang]])
whatis([[Version : 16.0.0-gfortran14.1.0]])
whatis([[Target : m2]])
whatis([[Short description : Use built-in clang for C/C++, and gfortran for Fortran.]])
whatis([[Configure options : --with-pkgversion=Spack GCC --with-bugurl=https://github.com/spack/spack/issues --disable-multilib --enable-languages=c,c++,fortran --disable-nls --disable-canonical-system-headers --with-system-zlib --with-zstd-include=/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/zstd-1.5.6-qgzwdavw3b3q35p2jt7qu4yb5kmkbskr/include --with-zstd-lib=/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/zstd-1.5.6-qgzwdavw3b3q35p2jt7qu4yb5kmkbskr/lib --with-ld=/Library/Developer/CommandLineTools/usr/bin/ld-classic --disable-bootstrap --with-mpfr-include=/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/mpfr-4.2.1-k6l2mohdprx77kyqgknyfgyyv7tfi5nh/include --with-mpfr-lib=/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/mpfr-4.2.1-k6l2mohdprx77kyqgknyfgyyv7tfi5nh/lib --with-gmp-include=/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/gmp-6.3.0-xt3cjmh7dehpxzqwjklftlml6vdqica2/include --with-gmp-lib=/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/gmp-6.3.0-xt3cjmh7dehpxzqwjklftlml6vdqica2/lib --with-mpc-include=/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/mpc-1.3.1-r4htrqxnrh4ia7hj5na3uxxihujzcgqf/include --with-mpc-lib=/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/mpc-1.3.1-r4htrqxnrh4ia7hj5na3uxxihujzcgqf/lib --without-isl --with-native-system-header-dir=/usr/include --with-sysroot=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk --with-libiconv-prefix=/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/libiconv-1.17-6735mlktagraawr36ttz4rzm6i5p6lis --with-stage1-ldflags=-Wl,-rpath,/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/gcc-14.1.0-yxln5vjswuutfxtzpq4tf5yk7iuhin4a/lib -Wl,-rpath,/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/gcc-14.1.0-yxln5vjswuutfxtzpq4tf5yk7iuhin4a/lib64 -Wl,-rpath,/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/gmp-6.3.0-xt3cjmh7dehpxzqwjklftlml6vdqica2/lib -Wl,-rpath,/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/libiconv-1.17-6735mlktagraawr36ttz4rzm6i5p6lis/lib -Wl,-rpath,/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/mpc-1.3.1-r4htrqxnrh4ia7hj5na3uxxihujzcgqf/lib -Wl,-rpath,/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/mpfr-4.2.1-k6l2mohdprx77kyqgknyfgyyv7tfi5nh/lib -Wl,-rpath,/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/zlib-ng-2.2.1-n5joj3sdmckf3f545vscegafl34xo6rz/lib -Wl,-rpath,/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/zstd-1.5.6-qgzwdavw3b3q35p2jt7qu4yb5kmkbskr/lib --with-boot-ldflags=-Wl,-rpath,/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/gcc-14.1.0-yxln5vjswuutfxtzpq4tf5yk7iuhin4a/lib -Wl,-rpath,/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/gcc-14.1.0-yxln5vjswuutfxtzpq4tf5yk7iuhin4a/lib64 -Wl,-rpath,/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/gmp-6.3.0-xt3cjmh7dehpxzqwjklftlml6vdqica2/lib -Wl,-rpath,/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/libiconv-1.17-6735mlktagraawr36ttz4rzm6i5p6lis/lib -Wl,-rpath,/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/mpc-1.3.1-r4htrqxnrh4ia7hj5na3uxxihujzcgqf/lib -Wl,-rpath,/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/mpfr-4.2.1-k6l2mohdprx77kyqgknyfgyyv7tfi5nh/lib -Wl,-rpath,/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/zlib-ng-2.2.1-n5joj3sdmckf3f545vscegafl34xo6rz/lib -Wl,-rpath,/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/zstd-1.5.6-qgzwdavw3b3q35p2jt7qu4yb5kmkbskr/lib -static-libstdc++ -static-libgcc --with-build-config=spack]])

help([[Name   : apple-clang]])
help([[Version: 16.0.0-gfortran14.1.0]])
help([[Target : m2]])
help()
help([[Use built-in clang for C/C++, and gfortran for Fortran.]])

-- Services provided by the package
family("compiler")

-- Loading this module unlocks the path below unconditionally
prepend_path("MODULEPATH", "/Users/sacks/spack/share/spack/lmod/darwin-sonoma-aarch64/apple-clang/16.0.0-gfortran14.1.0")



prepend_path("DYLD_FALLBACK_LIBRARY_PATH", "/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/gcc-14.1.0-yxln5vjswuutfxtzpq4tf5yk7iuhin4a/lib", ":")
prepend_path("PATH", "/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/gcc-14.1.0-yxln5vjswuutfxtzpq4tf5yk7iuhin4a/bin", ":")
prepend_path("MANPATH", "/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/gcc-14.1.0-yxln5vjswuutfxtzpq4tf5yk7iuhin4a/share/man", ":")
prepend_path("CMAKE_PREFIX_PATH", "/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/gcc-14.1.0-yxln5vjswuutfxtzpq4tf5yk7iuhin4a/.", ":")
setenv("CC", "/usr/bin/clang")
setenv("FC", "/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/gcc-14.1.0-yxln5vjswuutfxtzpq4tf5yk7iuhin4a/bin/gfortran")
setenv("F77", "/Users/sacks/spack/opt/spack/darwin-sonoma-m2/apple-clang-16.0.0/gcc-14.1.0-yxln5vjswuutfxtzpq4tf5yk7iuhin4a/bin/gfortran")
append_path("MANPATH", "", ":")

Note that, when installing new versions of gcc/gfortran, or doing something that causes significant changes in the spack-generated module files, I'll need to recreate this manually-created module file.

Installing NetCDF

First, with clang-gfortran with openmpi:

spack install netcdf-c %[email protected] ^openmpi
spack install netcdf-fortran %[email protected] ^openmpi

and with mpich (note that this also installs a new hdf5, since hdf5 depends on the mpi library):

spack install netcdf-c %[email protected] ^mpich
spack install netcdf-fortran %[email protected] ^mpich

and without mpi (note that we need to explicitly specify ^hdf5~mpi to avoid reusing an already-built hdf5 library that depends on mpi; this forces creating a new version of hdf5 without mpi):

spack install netcdf-c %[email protected] ~mpi ^hdf5~mpi
spack install netcdf-fortran %[email protected] ^netcdf-c~mpi

And then similar for gcc (note that I needed to explicitly ask for versions of dependencies that were built using gcc -- otherwise, spack tried to use dependencies like openmpi that were built using clang-gfortran; this caused problems in the paths where the netcdf modules were created, and might have caused other problems as well):

spack install netcdf-c %[email protected] ^openmpi%[email protected]
spack install netcdf-fortran %[email protected] ^openmpi%[email protected]
spack install netcdf-c %[email protected] ^mpich%[email protected]
spack install netcdf-fortran %[email protected] ^mpich%[email protected]
spack install netcdf-c %[email protected] ~mpi ^hdf5%[email protected]~mpi
spack install netcdf-fortran %[email protected] ^netcdf-c%[email protected]~mpi

Installing some python things

Python itself was already installed earlier (as a dependency of something else), but the ESMF testing needs some other things:

spack install py-pip
spack install py-pyyaml

This led to duplications of py-pip; I got rid of one with spack uninstal /bqgtfvr (to uninstall the one I had installed manually, just leaving the one that got installed in the course of installing pyyaml) (so maybe the first spack install py-pip was unnecessary).

Note that I now have two python modules, but the py-pip and py-pyyaml modules are built on python/3.11.9, so that one gets loaded when I load py-pip or py-pyyaml.

Installing other convenience software

emacs

Installed emacs with:

spack install emacs

Then added emacs to the core_specs section of .spack/modules.yaml.

Installing conda

I had trouble installing conda through spack (I tried both miniconda3 and anaconda3):

Error: failed to concretize `miniconda3` for the following reasons:
 1. Cannot select a single "version" for package "miniconda3"

(specifying a specific version with @ doesn't solve this).

So instead installing miniconda via https://docs.anaconda.com/miniconda/install/#quick-command-line-install

Then ran:

source ~/miniconda3/bin/activate
conda init --all

I'm also loading the necessary python modules in my .bash_profile. (We can't load them in the ESMF test scripts, because we need pyyaml available at the time when we start up these scripts.)

So my .bash_profile now looks like this:

# Spack stuff
. ${HOME}/spack/share/spack/setup-env.sh
. $(spack location -i lmod)/lmod/lmod/init/bash
module use ${SPACK_ROOT}/share/spack/lmod/darwin-sonoma-aarch64/Core/
module use ${HOME}/modules

# Load some modules
module load py-pip/23.1.2 py-pyyaml/6.0.2 python/3.11.9

# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/Users/sacks/miniconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
    eval "$__conda_setup"
else
    if [ -f "/Users/sacks/miniconda3/etc/profile.d/conda.sh" ]; then
        . "/Users/sacks/miniconda3/etc/profile.d/conda.sh"
    else
        export PATH="/Users/sacks/miniconda3/bin:$PATH"
    fi
fi
unset __conda_setup
# <<< conda initialize <<<

Also running:

conda config --set auto_activate_base false

Known issues

Issues with duplicate rpaths

When building an application with a mix of gfortran and clang, where clang is used as the final linker, under some circumstances clang produces an error, duplicate LC_RPATH. Specifically, I have encountered this when an intermediate library is created with gfortran and then clang is used to link this library with other libraries and/or object files. e.g., see https://github.com/spack/spack/issues/46506#issuecomment-2492650588.

In ESMF testing, this only shows up when using gfortranclang with openmpi (because, with this combination, the trace-related library is linked with the Fortran compiler rather than the C++ compiler, but then the final application is linked using the C++ compiler, as is the case in general with gfortranclang). For now I am avoiding this issue by setting ESMF_TRACE_LIB_BUILD=OFF for this combination.