Introduce version files and their usage in develop branch

[KateFriedman-NOAA]

Description

Introduce versions (.ver) files in a new /versions folder and update modulefiles to use module version variables when loading modules. This is related to work that occurred for the operations branch during the WCOSS2 port. See issues #639 and #398.

Will also take care of issue #1625 in the initial PR for this.

Requirements

• Alignment of module versions across components
• Availability of needed module versions via hpc-stack on supported tier-1 platforms

Acceptance Criteria (Definition of Done)

• new /versions folder exists and contains .ver files
• build.ver and run.ver exist and support loading correct module versions at buildtime/runtime
• additional $target.ver files for supported tier-1 platforms created and override need module versions on those platforms
• clean out old/outdated/unneeded version files (e.g. env/gfs.ver)

Dependencies

• Issue Convert modulefiles to LUA format (develop) #670 complete - Need modulefiles converted to LUA format.

Other TO-DOs

• Roll in updates from develop.
• Roll in update to obsproc_ver and obsproc_run_ver from develop (v1.2.0).

[KateFriedman-NOAA]

Created branch for this work: https://github.com/NOAA-EMC/global-workflow/tree/feature/version-files

[KateFriedman-NOAA]

Due to complications related to aligning component versions in the develop branch, this work to introduce version files is now ON HOLD until further work and discussions can occur.

[KateFriedman-NOAA]

@aerorahul @WalterKolczynski-NOAA Question about g2tmpl versions....on WCOSS2 we're currently using g2tmpl 1.9.1 but 1.10.0 everywhere else. Also, 1.10.2 is available on some but not all platforms in the stacks we're currently using in develop. The ufs-weather-model and UPP are using a mixture of 1.10.0 and 1.10.2 in their build modulefiles depending on which platform.

Any objections to moving to 1.10.0 on WCOSS2 as well for now and then reevaluate going to 1.10.2 with EPIC/spack? The EPIC stacks have both 1.10.0 and 1.10.2.

I ran with 1.10.0 in my testing on WCOSS2 and I'm getting bitwise results against develop so it looks like we can move to 1.10.0 on WCOSS2 and align with the R&Ds. We'll also be moving from wgrib2 2.0.7 to 2.0.8 on WCOSS2 in this version file introduction (another alignment of versions across machines).

Reminder, the first round PR for introducing version files into develop will not force version for build, only for runtime.

[WalterKolczynski-NOAA]

Fine with me

[KateFriedman-NOAA]

Tests run on Hera, Orion, and WCOSS2 using feature/versions-files produced bit-wise identical results to control tests using develop. A comparison of loaded modules and their versions shows expected diffs with g2tmpl and wgrib2 on WCOSS2 (moving g2tmpl from 1.9.1 to 1.10.1 and wgrib2 from 2.0.7 to 2.0.8 to align versions across machines).

Hera

develop:

Currently Loaded Modules:
  1) hpss/hpss        11) nco/4.9.1        21) intel/18.0.5.274
  2) gempak/7.4.2     12) prod_util/1.2.2  22) hpc-intel/18.0.5.274
  3) ncl/6.6.2        13) grib_util/1.2.2  23) miniconda3/4.6.14
  4) jasper/2.0.25    14) g2tmpl/1.10.0    24) hpc-miniconda3/4.6.14
  5) png/1.6.35       15) ncdiag/1.0.0     25) ufswm/1.0.0
  6) grib_api/1.26.1  16) crtm/2.4.0       26) met/9.1
  7) cdo/1.9.5        17) wgrib2/2.0.8     27) metplus/3.1
  8) R/3.5.0          18) prepobs/1.0.1    28) module_base.hera
  9) hdf5/1.10.6      19) fit2obs/1.0.0    29) impi/2018.0.4
 10) netcdf/4.7.4     20) hpc/1.2.0        30) hpc-impi/2018.0.4

Test using feature/versions-files (force versions at runtime via version files):

Currently Loaded Modules:
  1) hpss/hpss        11) nco/4.9.1        21) intel/18.0.5.274
  2) gempak/7.4.2     12) prod_util/1.2.2  22) hpc-intel/18.0.5.274
  3) ncl/6.6.2        13) grib_util/1.2.2  23) miniconda3/4.6.14
  4) jasper/2.0.25    14) g2tmpl/1.10.0    24) hpc-miniconda3/4.6.14
  5) png/1.6.35       15) ncdiag/1.0.0     25) ufswm/1.0.0
  6) grib_api/1.26.1  16) crtm/2.4.0       26) met/9.1
  7) cdo/1.9.5        17) wgrib2/2.0.8     27) metplus/3.1
  8) R/3.5.0          18) prepobs/1.0.1    28) module_base.hera
  9) hdf5/1.10.6      19) fit2obs/1.0.0    29) impi/2018.0.4
 10) netcdf/4.7.4     20) hpc/1.2.0        30) hpc-impi/2018.0.4

Orion

develop:

Currently Loaded Modules:
  1) gempak/7.5.1     12) wgrib2/2.0.8           23) netcdf/4.7.2
  2) ncl/6.6.2        13) prepobs/1.0.1          24) udunits/2.2.26
  3) jasper/2.0.25    14) fit2obs/1.0.0          25) antlr/2.7.7
  4) zlib/1.2.11      15) hpc/1.2.0              26) nco/4.8.1
  5) png/1.6.35       16) hpc-intel/2018.4       27) wgrib/2.0.8
  6) cdo/1.9.5        17) miniconda3/4.6.14      28) metplus/3.1
  7) prod_util/1.2.2  18) hpc-miniconda3/4.6.14  29) module_base.orion
  8) grib_util/1.2.2  19) ufswm/1.0.0            30) impi/2018.4
  9) g2tmpl/1.10.0    20) met/9.1                31) hpc-impi/2018.4
 10) ncdiag/1.0.0     21) intel/2018.4
 11) crtm/2.4.0       22) hdf5/1.10.5

Test using feature/versions-files (force versions at runtime via version files):

Currently Loaded Modules:
  1) gempak/7.5.1     12) wgrib2/2.0.8           23) netcdf/4.7.2
  2) ncl/6.6.2        13) prepobs/1.0.1          24) udunits/2.2.26
  3) jasper/2.0.25    14) fit2obs/1.0.0          25) antlr/2.7.7
  4) zlib/1.2.11      15) hpc/1.2.0              26) nco/4.8.1
  5) png/1.6.35       16) hpc-intel/2018.4       27) wgrib/2.0.8
  6) cdo/1.9.5        17) miniconda3/4.6.14      28) metplus/3.1
  7) prod_util/1.2.2  18) hpc-miniconda3/4.6.14  29) module_base.orion
  8) grib_util/1.2.2  19) ufswm/1.0.0            30) impi/2018.4
  9) g2tmpl/1.10.0    20) met/9.1                31) hpc-impi/2018.4
 10) ncdiag/1.0.0     21) intel/2018.4
 11) crtm/2.4.0       22) hdf5/1.10.5

WCOSS2

develop:

Currently Loaded Modules:
  1) craype-x86-rome     (H)  12) gempak/7.14.1    23) bufr_dump/1.0.0
  2) libfabric/1.11.0.0. (H)  13) perl/5.32.0      24) util_shared/1.4.0
  3) craype-network-ofi  (H)  14) libjpeg/9c       25) g2tmpl/1.9.1
  4) envvar/1.0               15) udunits/2.2.28   26) ncdiag/1.0.0
  5) PrgEnv-intel/8.1.0       16) gsl/2.7          27) crtm/2.4.0
  6) craype/2.7.13            17) cdo/1.9.8        28) wgrib2/2.0.7
  7) intel/19.1.3.304         18) hdf5/1.10.6      29) prepobs/1.0.1
  8) cray-mpich/8.1.9         19) netcdf/4.7.4     30) fit2obs/1.0.0
  9) cray-pals/1.0.17         20) nco/4.7.9        31) module_base.wcoss2
 10) cfp/2.0.4                21) prod_util/2.0.9
 11) python/3.8.6             22) grib_util/1.2.3

Test using feature/versions-files (force versions at runtime via version files):

Currently Loaded Modules:
  1) craype-x86-rome     (H)  12) gempak/7.14.1    23) bufr_dump/1.0.0
  2) libfabric/1.11.0.0. (H)  13) perl/5.32.0      24) util_shared/1.4.0
  3) craype-network-ofi  (H)  14) libjpeg/9c       25) g2tmpl/1.10.0
  4) envvar/1.0               15) udunits/2.2.28   26) ncdiag/1.0.0
  5) PrgEnv-intel/8.1.0       16) gsl/2.7          27) crtm/2.4.0
  6) craype/2.7.13            17) cdo/1.9.8        28) wgrib2/2.0.8
  7) intel/19.1.3.304         18) hdf5/1.10.6      29) prepobs/1.0.1
  8) cray-mpich/8.1.9         19) netcdf/4.7.4     30) fit2obs/1.0.0
  9) cray-pals/1.0.17         20) nco/4.7.9        31) module_base.wcoss2
 10) cfp/2.0.4                21) prod_util/2.0.9
 11) python/3.8.6             22) grib_util/1.2.3

Unable to test on S4. Have not tested on Jet. Review of changes to those machine modulefiles shows that versions should be left the same in this work.