Cobra installation instructions

Here, instructions are given for installing simsopt from source on the IPP cluster Cobra. These steps worked as of 18 November 2022. Similar steps should work on the IPP cluster Raven. Note that at any stage, you can run conda list or pip list to list the conda and/or python packages that are installed in your current conda virtual environment.

First, make sure the anaconda/3/2020.02 module is loaded.

Next, we will set up a conda virtual environment with simsopt and all of its dependencies. As many packages as possible are installed via conda, and those that are not available on conda are installed via pip. Anywhere in your file system, create a file named simsopt.yml with the following contents:

name: simsopt
channels:
  - conda-forge
dependencies:
  - python=3.10
  - numpy
  - scipy
  - cmake
  - ninja
  - pybind11
  - jax
  - jaxlib
  - scikit-build
  - matplotlib
  - monty
  - nptyping
  - Deprecated
  - randomgen
  - ruamel.yaml
  - sympy
  - h5py
  - f90nml
  - pyevtk
  - setuptools_scm

Now run

conda env create -f simsopt.yml

It will take a few minutes for conda to download the packages for the virtual environment.

If you have not previously run conda init, do this now. You will be instructed to log out and log in again. Do this.

Now, to activate the conda virtual environment that we will use for simsopt, enter

conda activate simsopt

Your prompt should now begin with the conda environment in parentheses, as in

(/u/mlan/conda-envs/simsopt) mlan@cobra03:~>

(but with your username in place of mlan.) You may wish to add conda activate simsopt to your .bashrc script so you do not need to manually type it every time you log in.

At this point, ensure the following modules are loaded:

Currently Loaded Modulefiles:
  1) intel/19.1.3         3) mkl/2020.4           5) netcdf-mpi/4.4.1     7) gcc/9
  2) impi/2019.9          4) hdf5-mpi/1.8.22      6) anaconda/3/2020.02

Even though the Intel compiler will be used, it is necessary to load the gcc module, as the Intel C++ compiler relies on some gnu libraries. There is a known issue with VMEC and gcc/10, so use gcc/9 for now.

Next, we must build mpi4py using the system's MPI. To do this, run

env MPICC=mpiicc pip install --no-cache-dir mpi4py

(The --no-cache-dir option is usually unnecessary, but it ensures that a clean build is performed in case any temporary files are left from previous unsuccessful build attempts.)

Now navigate to your copy of the simsopt repostitory. (If you haven't already downloaded it, this can be done with git clone https://github.com/hiddenSymmetries/simsopt.git.) Now run

CC=icc CXX=icpc pip install -v -e .

(The -v flag causes more diagnostic output to be printed, and the -e flag makes the installation "editable", so changes you make to the python source will be immediately reflected in the installed simsopt module.) It will take a minute to compile the compiled parts of simsopt. Warnings about unknown option '-ffp-contract=fast' can be ignored. If anything goes wrong, it is sometimes necessary to delete the build directory before trying again, to ensure a clean build.

At this point, simsopt should be installed. To check the installation, you can try the following:

python -c "import simsopt; print(simsopt.__version__)"

Note that on Cobra, simsopt modules that use MPI can only be imported in python from a compute node (either via a batch script or using srun in an interactive session), not from a login node. The reason is that Cobra does not allow MPI to be initialized from a login node. If you try, python will exit with an error like this:

Abort(1091087) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
MPIR_Init_thread(136): 
MPID_Init(950).......: 
MPIR_pmi_init(168)...: PMI2_Job_GetId returned 14

To check simsopt components that use MPI, you can log in to cobra-i.mpcdf.mpg.de (cobra03-cobra06) and try something like the following:

srun -t 1 -n 1 --mem 256M -p interactive python -c "import simsopt.util.mpi; print('success')"

If you do not require VMEC, (for instance if you are doing stage-2 coil optimization), you can stop here.

If you do want to use VMEC inside a simsopt optimization, first our fork of f90wrap must be installed using

pip install -U git+https://github.com/zhucaoxiang/f90wrap

Next, clone the repository for the python-wrapped VMEC using git clone https://github.com/hiddenSymmetries/VMEC2000.git, and navigate to the cloned repository using cd VMEC2000. Copy the Cobra configuration file using

cp cmake/machines/cobra_intel.json cmake_config_file.json

Or, equivalently, you can edit the cmake_config_file.json file to look as follows:

{
    "cmake_args": [
        "-DCMAKE_C_COMPILER=mpiicc",
        "-DCMAKE_CXX_COMPILER=mpiicpc",
        "-DCMAKE_Fortran_COMPILER=mpiifort",
        "-DNETCDF_INC_PATH=/mpcdf/soft/SLE_12/packages/skylake/netcdf-mpi/intel_19.1.1-19.1.1-impi_2019.7-2019.7.217/4.4.1/include",
        "-DNETCDF_LIB_PATH=/mpcdf/soft/SLE_12/packages/skylake/netcdf-mpi/intel_19.1.1-19.1.1-impi_2019.7-2019.7.217/4.4.1/lib",
        "-DSCALAPACK_LIB_DIR=/mpcdf/soft/SLE_12/packages/x86_64/intel_parallel_studio/2020.1/mkl/lib/intel64",
        "-DSCALAPACK_LIB_NAME=mkl_scalapack_lp64",
        "-DBLACS_LIB_DIR=/mpcdf/soft/SLE_12/packages/x86_64/intel_parallel_studio/2020.1/mkl/lib/intel64",
        "-DBLACS_LIB_NAME=mkl_blacs_intelmpi_lp64"]
}

Now run

python setup.py install

If anything goes wrong with the compilation and you need to re-build the vmec module, you may want to delete the _skbuild directory, so a clean build is performed.

If you would like to use booz_xform, it can be installed via

pip install booz_xform

Finally, before importing the vmec python module, you must add the MKL and NetCDF libraries to the path:

export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${MKL_HOME}/lib/intel64:${NETCDF_HOME}/lib

(You may wish to add the line above to your .bashrc file for convenience.) Now you should be able to import the vmec python module:

srun -t 1 -n 1 --mem 256M -p interactive python -c "import vmec; print('success')"