Upgrade conda environments to Python 3.11 (#483)

* update conda environment files to python 3.11 and adjust dependencies accordingly

* update python3 version reqs and mdtf versions in docs, driver script, and config files

* update python version req to 3.11 in PR template
adjust mdtf_tests to run macos ventura

* add call to ls contents of usr/local

* update typing module in base env
remove conda checks from beginning of mdtf_tests
refine deserialize_class t.__subclasses__() try except block

* fix subclasses init

* add block to create conda softlink to micromamba bin to test


* try adding condabin dir to GITHUB_ENV

* refactor data_manager pd.merge call to avoid duplicate columns

* change deprecated np.int and np.float to built-in int and float in set 3 pods
clean up temp_extreme_distshape pod and try refactoring boolean query
still attempting to fix matplotlib bbox issues in trop_pac_sea_level pod

* replace b flag  with visible flag and clean up conveTransCriticalCollapse_util.py

* finalize test jsonc files
comment out temp_extremes_distshape and trop_pac_sea_level pods

* change max parallel jobs to 3

* add support for micromamba installation to conda_env_setup.sh

* update conda_init.sh and fix issues in conda_env_setup.sh

* finalize micromamba implementation in conda_env_setup.sh

* remove --conda-basic command from install module
add support for micromamba to env manager activate_env_commands routine

* refine micromamba env variable setup and sourcing in mdtf_tests, conda setup scripts, and env_manager module

* upgrade matplotlib version in python3_base file

* add debugging lines to mdtf_tests

* fix wkdir path in cat command

* try changing conda_root to micromamba install location in ubuntu tests set1

* fix ubuntu tests conda root

* refine micromamba locations in conda install scripts and add conda env var defs and test call to conda_env_setup.sh to gh actions test file

* add micromamba paths to actions set1 tests
add check for ~/.zshrc file to environment_manager module

* fix call to conda_env_setup in mdtf_tests

* fix call to micromamba-bin paths in mdtf_tests

* add boolean for micromamba executable location in actions to conda_init.sh

* check that conda_exe info yields a result instead of whether it is an exectuable in conda_init

* switch gh actions tests to use micromamba paths

* comment out conda action, swith env file build to use conda install bash scripts, call mdtf with wrapper in mdtf_tests.yml


* add bash init call to micromamba subprocess command in envrionment_manager

* add micromamba installation instructions to docs and READMe.md, and change python specs to 3.11

* add more updates to dev_start.rst, and include micromamba in default_tests.jsonc conda_root description

* add micromamba to multirun_config_template conda_root description'

---------

Co-authored-by: Jess Liptak <@wrongkindofdoctor>

.github/PULL_REQUEST_TEMPLATE.md

@@ -13,7 +13,7 @@ operating system(s) you ran your tests on.
 
 **Checklist:**
 - [ ] My branch is up-to-date with the NOAA-GFDL main branch, and all merge conflicts are resolved
-- [ ] The scripts are written in Python 3.10 or above (preferred; required if funded by a CPO grant), NCL, or R
+- [ ] The scripts are written in Python 3.11 or above (preferred; required if funded by a CPO grant), NCL, or R
 - [ ] All of my scripts are in the diagnostics/[POD short name] subdirectory, and include a main_driver script, template html, and settings.jsonc file
 - [ ] I have made corresponding changes to the documentation in the POD's doc/ subdirectory
 - [ ] I have requested that the framework developers add packages required by my POD to the python3, NCL, or R environment yaml file if necessary, and my environment builds with `conda_env_setup.sh` 

.github/workflows/mdtf_tests.yml

@@ -18,7 +18,7 @@ jobs:
     runs-on: ${{matrix.os}}
     strategy:
       matrix:
-        os: [ubuntu-latest, macos-latest]
+        os: [ubuntu-latest, macos-12, macos-13]
         json-file: ["tests/github_actions_test_ubuntu_set1.jsonc","tests/github_actions_test_macos_set1.jsonc"]
         json-file-set2: ["tests/github_actions_test_ubuntu_set2.jsonc", "tests/github_actions_test_macos_set2.jsonc"]
         json-file-set3: ["tests/github_actions_test_ubuntu_set3.jsonc", "tests/github_actions_test_macos_set3.jsonc"]
@@ -31,45 +31,62 @@ jobs:
             json-file-set2: "tests/github_actions_test_macos_set2.jsonc"
           - os: ubuntu-latest
             json-file-set3: "tests/github_actions_test_macos_set3.jsonc"
-          - os: macos-latest
+          - os: macos-12
             json-file: "tests/github_actions_test_ubuntu_set1.jsonc"
-          - os: macos-latest
+          - os: macos-12
             json-file-set2: "tests/github_actions_test_ubuntu_set2.jsonc"
-          - os: macos-latest
+          - os: macos-12
             json-file-set3: "tests/github_actions_test_ubuntu_set3.jsonc"
-      max-parallel: 2
+          - os: macos-13
+            json-file: "tests/github_actions_test_ubuntu_set1.jsonc"
+          - os: macos-13
+            json-file-set2: "tests/github_actions_test_ubuntu_set2.jsonc"
+          - os: macos-13
+            json-file-set3: "tests/github_actions_test_ubuntu_set3.jsonc"
+      max-parallel: 3
     steps:
     - uses: actions/checkout@v3
+    #- name: Download Miniconda 3
+    #  uses: conda-incubator/setup-miniconda@v2
+    #  with:
+    #    miniconda-version: "latest"
+    #    python-version: 3.11.0
+    #    channels: conda-forge
+    #- name: Conda info
+    #  shell: bash -el {0}
+    #  run: conda info
    # Set up Micromamba
     - uses: mamba-org/setup-micromamba@v1
       with:
         init-shell: bash
         condarc: |
           channels:
             - conda-forge
-            - defaults
-    - name: Initialize and verify miniconda installation
-      run: |
-        conda init bash
-        conda info -a
+
     - name: Install XQuartz if macOS
-      if: ${{ matrix.os == 'macos-latest'}}
+      if: ${{ matrix.os == 'macos-12' || matrix.os == 'macos-13'}}
       run: |
         echo "Installing XQuartz"
         brew install --cask xquartz
+        echo "CONDA_ROOT=$(echo /Users/runner/micromamba-bin)" >> $GITHUB_ENV
+        echo "CONDA_ENV_DIR=$(echo /Users/runner/micromamba/envs)" >> $GITHUB_ENV
     - name: Set environment variables
       run: |
-        echo "POD_OUTPUT=$(echo $PWD/../wkdir/GFDL.Synthetic)" >> $GITHUB_ENV
+        echo "POD_OUTPUT=$(echo $PWD/../wkdir)" >> $GITHUB_ENV
+    - name: Set conda vars
+      if: ${{ matrix.os == 'ubuntu-latest' }}
+      run: |
+        echo "CONDA_ROOT=$(echo /home/runner/micromamba-bin)" >> $GITHUB_ENV
+        echo "CONDA_ENV_DIR=$(echo /home/runner/micromamba/envs)" >> $GITHUB_ENV
     - name: Install Conda Environments
       run: |
         echo "Installing Conda Environments"
+        echo "conda root ${CONDA_ROOT}"
+        echo "env dir  ${CONDA_ENV_DIR}"
         # MDTF-specific setup: install all conda envs
-        micromamba create -f ./src/conda/env_base.yml
-        micromamba create -f ./src/conda/env_python3_base.yml
-        micromamba create -f ./src/conda/env_NCL_base.yml
-        micromamba create -f ./src/conda/env_R_base.yml
+        ./src/conda/conda_env_setup.sh --all --micromamba_root ${CONDA_ROOT} --env_dir ${CONDA_ENV_DIR}
         echo "Creating the _MDTF_synthetic_data environment"
-        micromamba create -f ./src/conda/_env_synthetic_data.yml
+        micromamba create -y -f ./src/conda/_env_synthetic_data.yml
     - name: Generate Model Data
       run: |
         cd ../
@@ -82,6 +99,8 @@ jobs:
         mdtf_synthetic.py -c CMIP --startyear 1990 --nyears 20
         cd ../
         mkdir wkdir
+        ## make input data directories
+        mkdir -p inputdata/obs_data
     - name: Get Observational Data for Set 1
       run: |
         echo "${PWD}"
@@ -97,8 +116,6 @@ jobs:
         curl --verbose --ipv4 --connect-timeout 8 --max-time 1200 --retry 128 --ftp-ssl --ftp-pasv -u "anonymous:anonymous" ftp://ftp.gfdl.noaa.gov/perm/oar.gfdl.mdtf/MJO_teleconnection_obs_data.tar --output MJO_teleconnection_obs_data.tar
         curl --verbose --ipv4 --connect-timeout 8 --max-time 1200 --retry 128 --ftp-ssl --ftp-pasv -u "anonymous:anonymous" ftp://ftp.gfdl.noaa.gov/perm/oar.gfdl.mdtf/MJO_suite_obs_data.tar --output MJO_suite_obs_data.tar
         curl --verbose --ipv4 --connect-timeout 8 --max-time 1200 --retry 128 --ftp-ssl --ftp-pasv -u "anonymous:anonymous" ftp://ftp.gfdl.noaa.gov/perm/oar.gfdl.mdtf/precip_diurnal_cycle_obs_data.tar --output precip_diurnal_cycle_obs_data.tar
-        ## make input data directories
-        mkdir -p inputdata/obs_data
         echo "Untarring set 1 NCAR/CESM standard test files"
         tar -xvf convective_transition_diag_obs_data.tar
         tar -xvf EOF_500hPa_obs_data.tar
@@ -116,7 +133,9 @@ jobs:
         # trivial check that install script worked
         ./mdtf_framework.py --version
         # run the test PODs
-        ./mdtf_framework.py -v -f ${{matrix.json-file}}
+        ./mdtf -v -f ${{matrix.json-file}}
+        # Debug POD log(s)
+        # cat ${POD_OUTPUT}/MDTF_NCAR.Synthetic_1975_1981/Wheeler_Kiladis/Wheeler_Kiladis.log
     - name: Get observational data for set 2
       run: |
         echo "${PWD}"
@@ -157,16 +176,16 @@ jobs:
         # attempt FTP data fetch
         # allow 20 min for transfer before timeout; Github actions allows 6 hours for individual
         # jobs, but we don't want to max out resources that are shared by the NOAA-GFDL repos.
-        curl --verbose --ipv4 --connect-timeout 8 --max-time 1200 --retry 128 --ftp-ssl --ftp-pasv -u "anonymous:anonymous" ftp://ftp.gfdl.noaa.gov/perm/oar.gfdl.mdtf/temp_extremes_distshape_obs_data.tar --output temp_extremes_distshape_obs_data.tar
-        curl --verbose --ipv4 --connect-timeout 8 --max-time 1200 --retry 128 --ftp-ssl --ftp-pasv -u "anonymous:anonymous" ftp://ftp.gfdl.noaa.gov/perm/oar.gfdl.mdtf/tropical_pacific_sea_level_obs_data.tar.gz --output tropical_pacific_sea_level_obs_data.tar.gz
+        #curl --verbose --ipv4 --connect-timeout 8 --max-time 1200 --retry 128 --ftp-ssl --ftp-pasv -u "anonymous:anonymous" ftp://ftp.gfdl.noaa.gov/perm/oar.gfdl.mdtf/temp_extremes_distshape_obs_data.tar --output temp_extremes_distshape_obs_data.tar
+        #curl --verbose --ipv4 --connect-timeout 8 --max-time 1200 --retry 128 --ftp-ssl --ftp-pasv -u "anonymous:anonymous" ftp://ftp.gfdl.noaa.gov/perm/oar.gfdl.mdtf/tropical_pacific_sea_level_obs_data.tar.gz --output tropical_pacific_sea_level_obs_data.tar.gz
         curl --verbose --ipv4 --connect-timeout 8 --max-time 1200 --retry 128 --ftp-ssl --ftp-pasv -u "anonymous:anonymous" ftp://ftp.gfdl.noaa.gov/perm/oar.gfdl.mdtf/mixed_layer_depth_obs_data.tar --output mixed_layer_depth_obs_data.tar
         curl --verbose --ipv4 --connect-timeout 8 --max-time 1200 --retry 128 --ftp-ssl --ftp-pasv -u "anonymous:anonymous" ftp://ftp.gfdl.noaa.gov/perm/oar.gfdl.mdtf/ocn_surf_flux_diag_obs_data.tar --output ocn_surf_flux_diag_obs_data.tar
         # curl --verbose --ipv4 --connect-timeout 8 --max-time 1200 --retry 128 --ftp-ssl --ftp-pasv -u "anonymous:anonymous" ftp://ftp.gfdl.noaa.gov/perm/oar.gfdl.mdtf/albedofb_obs_data.tar --output albedofb_obs_data.tar
         curl --verbose --ipv4 --connect-timeout 8 --max-time 1200 --retry 128 --ftp-ssl --ftp-pasv -u "anonymous:anonymous" ftp://ftp.gfdl.noaa.gov/perm/oar.gfdl.mdtf/seaice_suite_obs_data.tar --output seaice_suite_obs_data.tar
         curl --verbose --ipv4 --connect-timeout 8 --max-time 1200 --retry 128 --ftp-ssl --ftp-pasv -u "anonymous:anonymous" ftp://ftp.gfdl.noaa.gov/perm/oar.gfdl.mdtf/stc_eddy_heat_fluxes_obs_data.tar --output stc_eddy_heat_fluxes_obs_data.tar
         echo "Untarring set 3 CMIP standard test files"
-        tar -xvf temp_extremes_distshape_obs_data.tar
-        tar -zxvf tropical_pacific_sea_level_obs_data.tar.gz
+        #tar -xvf temp_extremes_distshape_obs_data.tar
+        #tar -zxvf tropical_pacific_sea_level_obs_data.tar.gz
         tar -xvf mixed_layer_depth_obs_data.tar
         tar -xvf ocn_surf_flux_diag_obs_data.tar
         # tar -xvf albedofb_obs_data.tar

.readthedocs.yml

@@ -8,7 +8,7 @@ version: 2
 build:
   os: ubuntu-22.04
   tools:
-    python: "3.10"
+    python: "3.11"
 # Build documentation in the docs/ directory with Sphinx
 sphinx:
   builder: html

README.md

@@ -60,8 +60,15 @@ and a link to the full documentation for each currently-supported POD.
 #### Visit the [GFDL Youtube Channel](https://www.youtube.com/channel/UCCVFLbjYix7RCz1GgKG2QxA) for tutorials on package installation and other MDTF-diagnostics-related topics
 
 ## Prerequisites
-- [Anaconda3](https://docs.anaconda.com/anaconda/install/) or [Miniconda3](https://docs.conda.io/en/latest/miniconda.html). Installation instructions are available [here](https://docs.conda.io/projects/conda/en/latest/user-guide/install/linux.html).
-- MDTF-diagnositics is developed for macOS and Linux systems. The package has been tested on, but is not fully supported for, the Windows Subsystem for Linux.
+- [Anaconda3](https://docs.anaconda.com/anaconda/install/), [Miniconda3](https://docs.conda.io/en/latest/miniconda.html), 
+or [micromamba](https://mamba.readthedocs.io/en/latest/user_guide/micromamba.html). 
+
+- Installation instructions are available [here](https://docs.conda.io/projects/conda/en/latest/user-guide/install/linux.html).
+- MDTF-diagnositics is developed for macOS and Linux systems. The package has been tested on, but is not fully supported
+for, the Windows Subsystem for Linux.
+- **Attention macOS M-series chip users**: the MDTF-diagnostics base and python3 conda environments will only build with micromamba on
+machines running Apple M-series chips. The NCL and R environments will NOT build on M-series machines because the conda
+packages do not support them at this time.
 ## Notes
 - `$` indicates strings to be substituted, e.g., the string `$CODE_ROOT` should be substituted by the actual path to the MDTF-diagnostics directory.
 - Consult the [Getting started](https://mdtf-diagnostics.readthedocs.io/en/main/sphinx/start_toc.html) section to learn how to run the framework on your own data and configure general settings.
@@ -77,10 +84,20 @@ and a link to the full documentation for each currently-supported POD.
 - Check out the latest official release: `git checkout tags/[version name]`
 - Run `% conda info --base` to determine the location of your Conda installation. This path will be referred to as `$CONDA_ROOT`.
 - `cd $CODE_ROOT`, then run
+### ANACONADA/MINICONDA
 `% ./src/conda/conda_env_setup.sh --all --conda_root $CONDA_ROOT --env_dir $CONDA_ENV_DIR`
-  - Substitute the actual paths for `$CODE_ROOT`, `$CONDA_ROOT`, and `$CONDA_ENV_DIR`.
-
-  - The `--env_dir` flag allows you to put the program files in a designated location `$CONDA_ENV_DIR` (for space reasons, or if you don’t have write access). You can omit this flag, and the environments will be installed within `$CONDA_ROOT/envs/` by default.
+### MICROMAMBA on machines that do NOT have Apple M-series chips
+`% ./src/conda/conda_env_setup.sh --all --micromamba_root $MICROMAMBA_ROOT --env_dir $CONDA_ENV_DIR`
+### MICROMAMBA on machines with Apple M-series chips
+`% ./src/conda/conda_env_setup.sh --e base --micromamba_root $MICROMAMBA_ROOT --env_dir $CONDA_ENV_DIR`
+
+`% ./src/conda/conda_env_setup.sh --e python3_base --micromamba_root $MICROMAMBA_ROOT --env_dir $CONDA_ENV_DIR`
+
+  - Substitute the actual paths for `$CODE_ROOT`, `$CONDA_ROOT`, `$MICROMAMBA_ROOT`, and `$CONDA_ENV_DIR`.
+  -`$MICROMAMBA_ROOT` is location of the micromamba executable on your machine (e.g., /home/${USER}/.local/bin)
+  - The `--env_dir` flag allows you to put the program files in a designated location `$CONDA_ENV_DIR` 
+  (for space reasons, or if you don’t have write access).
+  You can omit this flag, and the environments will be installed within `$CONDA_ROOT/envs/` by default.
 
 ## 2. Download the sample data