Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add separate environment files for micromamba installation #607

Merged
merged 2 commits into from
Jul 10, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,11 @@ for, the Windows Subsystem for Linux.
(for space reasons, or if you don’t have write access).
You can omit this flag, and the environments will be installed within `$CONDA_ROOT/envs/` by default.

#### NOTE: The micromamba environments may differ from the conda environments because of package compatibility discrepancies between solvers
The micromamba installation script only builds the **base** environment, and a limited version of the **python3_base**
enviroment that excludes the following packages and dependencies that may be required by the POD(s) you want to run:
- falwa
- gridfill
## 2. Download the sample data

Supporting observational data and sample model data are available via anonymous FTP at [ftp://ftp.cgd.ucar.edu/archive/mdtf](ftp://ftp.cgd.ucar.edu/archive/mdtf).
Expand Down
6 changes: 6 additions & 0 deletions doc/sphinx/start_install.rst
Original file line number Diff line number Diff line change
Expand Up @@ -227,6 +227,12 @@ Install all the package's conda environments with micromamba by running
These environments can be uninstalled by deleting their corresponding directories under <*CONDA_ENV_DIR*>
(or <*CONDA_ROOT*>/envs/).

.. note::
The micromamba environments may differ from the conda environments because of package compatibility discrepancies
between solvers. The micromamba installation script only builds the **base** environment, and a limited version of
the **python3_base** enviroment that excludes some packages and dependencies that may be required by the
POD(s) you want to run.

Location of the installed executable
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Expand Down
9 changes: 9 additions & 0 deletions src/conda/env_NCL_base_micromamba.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
name: _MDTF_NCL_base
channels:
- conda-forge
dependencies:
- python=3.11
# - openblas=0.3.4 # otherwise ncl 6.5 installs but doesn't function on macs
- ncl=6.6.2
- nco=5.1.7
- pyyaml=6.0.1
33 changes: 33 additions & 0 deletions src/conda/env_base_micromamba.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
name: _MDTF_base
# environment used by the framework itself
# use this file if building environment with micromamba
channels:
- conda-forge
dependencies:
# specify up to minor version number to provide a consistent environment for POD developers
# versions are current as of Jul 2020
# see https://docs.conda.io/projects/conda/en/latest/user-guide/concepts/pkg-specs.html
- python=3.11
- ghostscript
- numpy=1.25.2
- netCDF4=1.6.4
- cftime=1.6.2
- xarray=2023.8.0
# Note: newer versions of cf_xarray are causing issues with missing
# xarray dataset attributes. There seem to be modifications where
# ds.cf attributes are defined later in the process, and this clashes
# with the preprocessing procedures
- cf_xarray=0.8.4
- matplotlib=3.8.2
- pandas=2.1.0
- pint=0.22
- dask=2024.1.1
- ecgtools=2023.7.13
- cfunits=3.3.6
- intake=0.7.0
- intake-esm=2024.2.6
- subprocess32=3.5.4
- pyyaml=6.0.1
- click=8.1.7
- ghostscript=10.02.1
- pybind11=2.12.0
2 changes: 2 additions & 0 deletions src/conda/env_python3_base.yml
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,8 @@ dependencies:
- falwa==1.2.1
- cmocean
- regionmask
- git+https://github.com/csyhuang/hn2016_falwa
- git+https://github.com/ajdawson/gridfill
- git+https://github.com/raphaeldussin/static_downsampler
- git+https://github.com/jkrasting/xcompare
- git+https://github.com/raphaeldussin/xoverturning
Expand Down
49 changes: 49 additions & 0 deletions src/conda/env_python3_base_micromamba.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
name: _MDTF_python3_base
# use this file if building environment with Anaconda/Miniconda
channels:
- conda-forge # NOTE: critical to give highest priority to conda-forge
dependencies:
# specify up to minor version number to provide a consistent environment for POD developers
# see https://docs.conda.io/projects/conda/en/latest/user-guide/concepts/pkg-specs.

- python=3.11
- numpy=1.24.4
- scipy=1.11.2
- netCDF4=1.6.5
- cftime=1.6.2
- xarray=2024.1.1
- matplotlib=3.8.2
- pandas=2.1.0
- cartopy=0.22.0
- cython=3.0.2
- pint=0.22
- dask=2024.1.1
- numba=0.57.1
- scikit-learn=1.3.0
- xesmf=0.8.1
- esmf=8.4.2
- esmpy=8.4.2
- gsw=3.6.17
- metpy=1.5.1
#- enso_metrics=3.1.1 Required by pcmdi_metrics, but no support for python>3.10
#- pcmdi_metrics=3.0.2 Not used by current PODs-implement later
- h5py=3.9.0
- intake=0.7.0
- intake-xarray=0.7.0
- intake-esm=2024.2.6
- nc-time-axis=1.4.1
- pyyaml=6.0.1
- networkx=3.1
- jupyterlab=4.1.0
- bottleneck=1.3.8
- pip=23.3.1
- pip:
- cmocean
- regionmask
- git+https://github.com/ajdawson/gridfill
- git+https://github.com/raphaeldussin/static_downsampler
- git+https://github.com/jkrasting/xcompare
- git+https://github.com/raphaeldussin/xoverturning
- git+https://github.com/jkrasting/xwavelet
- git+https://github.com/jetesdal/xwmt
- git+https://github.com/raphaeldussin/om4labs
6 changes: 3 additions & 3 deletions src/conda/micromamba_env_setup.sh
Original file line number Diff line number Diff line change
Expand Up @@ -56,12 +56,12 @@ while (( "$#" )); do
;;
-a|--all)
# install all envs except dev environment
env_glob="env_!(dev).yml"
env_glob="env_!(dev)_micromamba.yml"
shift 1
;;
-e|--env)
# specify one env by name
env_glob="env_${2}.yml"
env_glob="env_${2}_micromamba.yml"
if [ ! -f "${script_dir}/${env_glob}" ]; then
echo "ERROR: ${script_dir}/${env_glob} not found."
exit 1
Expand All @@ -75,7 +75,7 @@ while (( "$#" )); do
;;
--all-dev)
# all envs, including dev
env_glob="env_*.yml"
env_glob="env_@(*micromamba|dev).yml"
shift 1
;;
-d|--env_dir)
Expand Down
Loading