Convert modulefiles to lua and hpc-stack (v16) #639

WalterKolczynski-NOAA · 2022-02-08T15:20:12Z

Description
The remaining (non-WCOSS2) modulefiles need to be converted to from TCL to lua on the v16 branch and accept version numbers. The updated modulefiles also need to load hpc-stack modules.

Requirements

Convert all module files from TCL to lua and allow versions to be set by sourcing a versions file beforehand.
Set modulefiles to load hpc-stack modules.

Acceptance Criteria (Definition of Done)

Workflow builds and works using no non-lua module files anywhere in global-workflow (including components) and accepts a single versions file provided by global-workflow.
Workflow builds and works using no non-hpc-stack modules anywhere in global-workflow (including components).

Companion issues in components:

GeorgeGayno-NOAA · 2022-02-08T15:51:05Z

FYI - the GFSv16.2.0 tag of UFS_UTILS is already 'lua' compliant on all non-WCOSS2 machines. See:
https://github.com/ufs-community/UFS_UTILS/releases/tag/ops-gfsv16.2.0 and ufs-community/UFS_UTILS#580

KateFriedman-NOAA · 2022-02-08T16:08:33Z

FYI - the GFSv16.2.0 tag of UFS_UTILS is already 'lua' compliant on all non-WCOSS2 machines. See: https://github.com/ufs-community/UFS_UTILS/releases/tag/ops-gfsv16.2.0 and ufs-community/UFS_UTILS#580

Thanks for confirming @GeorgeGayno-NOAA ! UFS_UTILS appeared to be the furthest ahead in the LUA game when we took a gander. Will just need the hpc-stack updates for Hera/Orion from you then, I see the issue you just opened, thanks!

- add the ".lua" extension to the Orion modulefiles Refs: NOAA-EMC#639

malloryprow · 2022-02-09T14:07:47Z

For EMC_verif-global:

On Hera, the anaconda version being loaded is anaconda/anaconda3-5.3.1. For the version of MET I need (/contrib/met/modulefiles/met/9.1 ), the prerequisites is anaconda/latest. Is that okay that I load anaconda/latest instead of anaconda/anaconda3-5.3.1 for EMC_verif-global?

On Orion: The version of MET needed (/apps/contrib/modulefiles/met/9.1 ) has prerequisites of intel/2020 and intelpython3/2020. For the global workflow, I believe intel/2018.4 and python/3.7.5 are the versions being used.

- cleaned out modulefiles that are no longer needed for OznMon/Radmon - remove errant workflow_utils.orion.lua (wrong branch) Refs: NOAA-EMC#639

Refs: NOAA-EMC#639

- add "setenv("myFC","ftn")" to the modulefiles for fbwndgfs and storm_reloc on WCOSS2 - needed to set FC on WCOSS2 now that other platforms have to set FC differently (e.g. mpiifort on Orion) Refs: NOAA-EMC#639

- update fbwndgfs makefile.GENERIC to use myFC variable - update build_tropcy_NEMS.sh to use myFC variable - also update build_tropcy_NEMS.sh to adjust JASPER_LIB variable and remove wcoss2 check for setting SIGIO_LIB4 and SIGIO_INC4 Refs: NOAA-EMC#639

- cleanup Orion, Hera, and Jet blocks to just set target and do a module purge - add $target.ver sourcing after build.ver sourcing Refs: NOAA-EMC#639

- update the gfs_bufr build and makefile to use hpc-stack library variables Refs: NOAA-EMC#639

- wrap $target.ver sourcing in block that checks if user is not on WCOSS2 and source if so - do not need or have wcoss2.ver file at this time Refs: NOAA-EMC#639

- adjust NETCDF_LDFLAGS_F flags in gaussian_sfcanl.fd/makefile.sh to flip -lnetcdf/-lnetcdff order, add -lhdf5_hl flag, and add -lz flag Refs: NOAA-EMC#639

- need to now source the target-specific version files before loading modulebase.$target.lua on Hera and Orion Refs: NOAA-EMC#639

- add NETCDF_LDFLAGS flags to successfully build regrid_nemsio Refs: NOAA-EMC#639

Refs: NOAA-EMC#639

- under the versions folder create new hera.ver and orion.ver - new target-specific version files will be used to set hpc-stack modules and versions, as well potentially override other versions as needed on the specific platforms Refs: NOAA-EMC#639

Refs: NOAA-EMC#639

- convert the contents of the newly renamed Hera modules to use LUA format - convert Hera modulefiles to load hpc-stack modules Refs: NOAA-EMC#639

- remove hdf5 and netcdf module loads from enkf_chgres_recenter modulefiles - reran builds to confirm Refs: NOAA-EMC#639

- remove nemsio and sigio modules from gfs_fbwndgfs build - reran builds to confirm Refs: NOAA-EMC#639

- was erroneously removed, added back into module_base.hera.lua - added companion hpss_ver to hera.ver Refs: NOAA-EMC#639

- create empty wcoss2.ver to go alongside other new machine-specific version files - update machine-setup.sh to load $target.ver on all machines now Refs: NOAA-EMC#639

- replace SIGIO_LIB4 with SIGIO_LIB - replace SIGIO_INC4 with SIGIO_INC - remove two lines that set "4" variables to non-"4" variables Refs: NOAA-EMC#639

MichaelLueken · 2022-02-15T10:29:47Z

The DA components for GFSv16.2.0 have been updated so that Hera and Orion modulefiles are both lua compliant. Please see the tag:

gfsda.v16.2.1 and issue #298 in the GSI repository.

KateFriedman-NOAA · 2022-02-15T17:08:34Z

The DA components for GFSv16.2.0 have been updated so that Hera and Orion modulefiles are both lua compliant. Please see the tag:
gfsda.v16.2.1 and issue #298 in the GSI repository.

Thanks @MichaelLueken-NOAA ! I have run build_gsi.sh successfully via my global-workflow branch on Orion. Please see the following log on Orion and confirm it did indeed build ok:
/work/noaa/global/kfriedma/git/dev_v16/sorc/logs/build_gsi.log

I tried the same build on Hera, it failed because I set the cmake version to be cmake/3.20.0 which comes from hpc-stack instead of the default cmake/3.16.1 available outside of the stack. Since cmake is loaded in the GSI modulefiles before the stack is loaded it didn't know about the cmake/3.20.0 version. See this log:
/scratch1/NCEPDEV/global/Kate.Friedman/git/dev_v16/sorc/logs/build_gsi.log

Can the cmake module load be moved after the stack module loads in the GSI modulefiles?

Also, a note on the new tag name (gfsda.v16.2.1)...since we could potentially have a GFSv16.2.1 in ops down the road could you retag with a different name? It's awkward for this update I understand but let me suggest gfsda.v16.2.0.1? I just wanna keep gfsda.v16.2.1 free for potential ops updates. Other components don't use the GFS version in their tag names so I haven't made similar comments to their new tags. Thanks!

- only available prod_util on Orion is prod_util/1.2.2 - add prod_util_ver=1.2.2 to orion.ver to override WCOSS2 version Refs: NOAA-EMC#639

- add overrides for prod_util and cmake versions to hera.ver - use available hpc-stack versions to override WCOSS2 defaults Refs: NOAA-EMC#639

- override WCOSS2 default for cmake_ver and use 3.22.1 on Orion Refs: NOAA-EMC#639

KateFriedman-NOAA · 2022-02-15T19:24:11Z

Merged #648 into dev_v16

MichaelLueken · 2022-02-16T15:12:43Z

The DA components for GFSv16.2.0 have been updated so that Hera and Orion modulefiles are both lua compliant. Please see the tag:
gfsda.v16.2.1 and issue #298 in the GSI repository.

Thanks @MichaelLueken-NOAA ! I have run build_gsi.sh successfully via my global-workflow branch on Orion. Please see the following log on Orion and confirm it did indeed build ok: /work/noaa/global/kfriedma/git/dev_v16/sorc/logs/build_gsi.log

I tried the same build on Hera, it failed because I set the cmake version to be cmake/3.20.0 which comes from hpc-stack instead of the default cmake/3.16.1 available outside of the stack. Since cmake is loaded in the GSI modulefiles before the stack is loaded it didn't know about the cmake/3.20.0 version. See this log: /scratch1/NCEPDEV/global/Kate.Friedman/git/dev_v16/sorc/logs/build_gsi.log

Can the cmake module load be moved after the stack module loads in the GSI modulefiles?

Also, a note on the new tag name (gfsda.v16.2.1)...since we could potentially have a GFSv16.2.1 in ops down the road could you retag with a different name? It's awkward for this update I understand but let me suggest gfsda.v16.2.0.1? I just wanna keep gfsda.v16.2.1 free for potential ops updates. Other components don't use the GFS version in their tag names so I haven't made similar comments to their new tags. Thanks!

@KateFriedman-NOAA Thanks for bringing these issues to my attention! I have updated modulefiles/modulefile.ProdGSI.hera.lua so that the cmake entry comes directly after setting up hpc-stack. I have tested that both cmake/3.16.1 and cmake/3.20.0 both successfully compile the DA components. After this, I removed the gfsda.v16.2.1 tag and created the gfsda.v16.2.0.1 tag.

Please let me know if you encounter any additional issues!

KateFriedman-NOAA · 2022-02-16T18:20:53Z

Thanks for bringing these issues to my attention! I have updated modulefiles/modulefile.ProdGSI.hera.lua so that the cmake entry comes directly after setting up hpc-stack. I have tested that both cmake/3.16.1 and cmake/3.20.0 both successfully compile the DA components. After this, I removed the gfsda.v16.2.1 tag and created the gfsda.v16.2.0.1 tag.

Thanks @MichaelLueken-NOAA ! It built ok on Hera, please see this log to confirm:

/scratch1/NCEPDEV/global/Kate.Friedman/git/dev_v16/sorc/logs/build_gsi.log_2

A few notes:

NCO asked us to move to w3emc/2.9.2 (which I am getting hpc-stack folks to install on Hera/Orion). I ran the build on Hera with export w3emc_ver=2.9.0 set via g-w versions/hera.ver since 2.9.2 isn't available yet. I see the GSI default is 2.7.3. Can you update that to 2.9.2?
Also, can you do the same cmake and w3emc updates to the Orion modulefile? I still see cmake and python loading before the stack modules. Best to load everything after the stack modules now, thanks!
Might need to update the same cmake and w3emc version for the WCOSS2 modulefile too.

MichaelLueken · 2022-02-16T19:01:01Z

Thanks @MichaelLueken-NOAA ! It built ok on Hera, please see this log to confirm:

/scratch1/NCEPDEV/global/Kate.Friedman/git/dev_v16/sorc/logs/build_gsi.log_2

A few notes:

NCO asked us to move to w3emc/2.9.2 (which I am getting hpc-stack folks to install on Hera/Orion). I ran the build on Hera with export w3emc_ver=2.9.0 set via g-w versions/hera.ver since 2.9.2 isn't available yet. I see the GSI default is 2.7.3. Can you update that to 2.9.2?

Also, can you do the same cmake and w3emc updates to the Orion modulefile? I still see cmake and python loading before the stack modules. Best to load everything after the stack modules now, thanks!

Might need to update the same cmake and w3emc version for the WCOSS2 modulefile too.

Thanks @KateFriedman-NOAA! I have updated the modulefiles for Hera, Orion, and WCOSS2 so that all modulel oads come after hpc-stack entries. I have also changed the w3emc default from 2.7.3 to 2.9.2. The tag is gfsda.v16.2.0.1.

KateFriedman-NOAA · 2022-02-16T20:57:14Z

I have updated the modulefiles for Hera, Orion, and WCOSS2 so that all modulel oads come after hpc-stack entries. I have also changed the w3emc default from 2.7.3 to 2.9.2. The tag is gfsda.v16.2.0.1.

Thanks @MichaelLueken-NOAA ! I recloned the updated gfsda.v16.2.0.1 tag on Hera, Orion, and WCOSS2. I ran the build on Orion and Hera with w3emc/2.9.1 (waiting on 2.9.2 install still). I built it on WCOSS2 as is. Please see the following build logs and confirm they indeed built ok:

Orion: /work/noaa/global/kfriedma/git/dev_v16/sorc/logs/build_gsi.log_2
Hera: /scratch1/NCEPDEV/global/Kate.Friedman/git/dev_v16/sorc/logs/build_gsi.log_3
WCOSS2: /lfs/h2/emc/global/noscrub/Kate.Friedman/git/dev_v16/sorc/logs/build_gsi.log

Assuming those built ok I will use the resulting execs when I start tests with the dev_v16 branch on Hera/Orion. Will report any runtime issues. Still waiting on library/module installs and other component updates (which are waiting on the same module installs).

MichaelLueken · 2022-02-17T13:36:06Z

Thanks @KateFriedman-NOAA! I have checked the build logs and there is one issue that I noted. For Hera, the version of anaconda that is being used is the default from the GSI's modulefile.ProdGSI.hera.lua file - anaconda/2.3.0. Since Hera uses anaconda instead of python, should the hera.ver file in dev_v16 include:

export anaconda_ver=anaconda3-5.3.1

KateFriedman-NOAA · 2022-02-17T14:34:31Z

Since Hera uses anaconda instead of python, should the hera.ver file in dev_v16 include:
export anaconda_ver=anaconda3-5.3.1

@MichaelLueken-NOAA Yes, good catch! I have added export anaconda_ver=anaconda3-5.3.1 to hera.ver and rerun the GSI build. Please see this log:
/scratch1/NCEPDEV/global/Kate.Friedman/git/dev_v16/sorc/logs/build_gsi.log_4

MichaelLueken · 2022-02-17T14:53:31Z

@KateFriedman-NOAA The new build log looks good!

KateFriedman-NOAA · 2022-02-17T14:54:55Z

@KateFriedman-NOAA The new build log looks good!

Awesome, thanks for confirming @MichaelLueken-NOAA ! I'll let you know how it goes when I start running tests on Hera/Orion with this updated tag. Waiting for modules and other components still.

KateFriedman-NOAA · 2022-02-17T20:42:01Z

For EMC_verif-global:
On Hera, the anaconda version being loaded is anaconda/anaconda3-5.3.1. For the version of MET I need (/contrib/met/modulefiles/met/9.1 ), the prerequisites is anaconda/latest. Is that okay that I load anaconda/latest instead of anaconda/anaconda3-5.3.1 for EMC_verif-global?

Using anaconda/latest would require at least the GSI and workflow to update (maybe others) and test with that version. I don't know how different they are.

On Orion: The version of MET needed (/apps/contrib/modulefiles/met/9.1 ) has prerequisites of intel/2020 and intelpython3/2020. For the global workflow, I believe intel/2018.4 and python/3.7.5 are the versions being used.

We're not using the 2020 versions yet and need to stick with the 2018 versions...at least regarding what we set via global-workflow. Is this a blocker for supporting METplus on Orion? Can we use the existing METplus versions we're using on Orion or has EMC_verif-global updated beyond that version too much now?

I'd prefer to avoid another layer of overrides for specific jobs but we may not have an option. :-/

malloryprow · 2022-02-18T13:10:18Z

@KateFriedman-NOAA I'd say this is a blocker as the MET and METplus module files on Hera and Orion require that the anaconda/latest, and intel/2020 and intelpython3/2020 (respectively) be loaded. These were what were used in the building of them.

I'm not sure why these specific versions were used for the build, but the MET team would need to rebuild on both Hera and Orion if we want them to be using the same versions as the global workflow. I know they are pretty busy right now pushing to get the next minor release out in the next few weeks.

KateFriedman-NOAA · 2022-02-18T16:14:31Z

I'd say this is a blocker as the MET and METplus module files on Hera and Orion require that the anaconda/latest, and intel/2020 and intelpython3/2020 (respectively) be loaded. These were what were used in the building of them.

Ok thanks for confirming @malloryprow . I'll need to speak with the workflow team about how we wanna support METplus for v16.2.0 on the R&D machines then. Stay tuned...thanks!

KateFriedman-NOAA · 2022-08-03T16:39:00Z

The bulk of the work for this issue is now complete. Remaining work to introduce version files into the develop branch is being tracked in issue #671.

…AA-EMC#639) * Make passing of PBL q tendency more general (not dependent on specific PBL scheme) for prog closure * Bugfix for convection (prog closure) and gravity wave drag (for stochastic perturbations) needed for GFS/GEFS prototypes

GeorgeGayno-NOAA mentioned this issue Feb 8, 2022

Update GFSv16.2.0 tag to use hpc-stack on Hera and Orion ufs-community/UFS_UTILS#629

Closed

junwang-noaa mentioned this issue Feb 8, 2022

GFSv16: update the module files on hera/orion ufs-community/ufs-weather-model#1039

Closed

KateFriedman-NOAA self-assigned this Feb 8, 2022

KateFriedman-NOAA added the maintenance Regular updates and maintenance work label Feb 8, 2022

WenMeng-NOAA mentioned this issue Feb 8, 2022

Update GFSV16 branch with hpc-stack on Hera and Orion NOAA-EMC/UPP#442

Closed

KateFriedman-NOAA mentioned this issue Feb 8, 2022

[INSTALL] Library modules to support GFSv16.2.0 on Hera/Orion NOAA-EMC/hpc-stack#379

Open

KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Feb 8, 2022

Rename Orion modulefiles for LUA format

90a69be

- add the ".lua" extension to the Orion modulefiles Refs: NOAA-EMC#639

MichaelLueken mentioned this issue Feb 9, 2022

Convert Hera/Orion modulefiles to lua for DA GFSv16.2.0 NOAA-EMC/GSI#298

Closed

KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Feb 9, 2022

Update Orion modulefiles to LUA format and hpc-stack

6e88009

Refs: NOAA-EMC#639

KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Feb 9, 2022

Cleanup machine-setup.sh

50cf445

- cleanup Orion, Hera, and Jet blocks to just set target and do a module purge - add $target.ver sourcing after build.ver sourcing Refs: NOAA-EMC#639

KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Feb 9, 2022

Update gfs_bufrsnd build for hpc-stack

4f063a9

- update the gfs_bufr build and makefile to use hpc-stack library variables Refs: NOAA-EMC#639

KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Feb 10, 2022

Update build_regrid_nemsio.sh with NETCDF_LDFLAGS

1af6dfa

- add NETCDF_LDFLAGS flags to successfully build regrid_nemsio Refs: NOAA-EMC#639

KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Feb 10, 2022

Rename Hera modulefiles to add .lua extension

55b9eb8

Refs: NOAA-EMC#639

KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Feb 10, 2022

Remove unneeded modulefile.grib_util.hera.lua

ed6024d

Refs: NOAA-EMC#639

KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Feb 11, 2022

Remove unneeded modules from fbwndgfs build

75fb6da

- remove nemsio and sigio modules from gfs_fbwndgfs build - reran builds to confirm Refs: NOAA-EMC#639

KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Feb 11, 2022

Add HPSS module load back into Hera module_base

7425fd6

- was erroneously removed, added back into module_base.hera.lua - added companion hpss_ver to hera.ver Refs: NOAA-EMC#639

KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Feb 15, 2022

Add prod_util_ver to orion.ver

7b3b96e

- only available prod_util on Orion is prod_util/1.2.2 - add prod_util_ver=1.2.2 to orion.ver to override WCOSS2 version Refs: NOAA-EMC#639

KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Feb 15, 2022

Add cmake_ver to orion.ver

238efed

- override WCOSS2 default for cmake_ver and use 3.22.1 on Orion Refs: NOAA-EMC#639

KateFriedman-NOAA changed the title ~~Convert modulefiles to lua (v16)~~ Convert modulefiles to lua and hpc-stack (v16) Feb 15, 2022

KateFriedman-NOAA mentioned this issue Feb 23, 2022

Support GFSv16.2.x (WCOSS2 port) on supported tier-1 platforms #665

Closed

KateFriedman-NOAA added this to the GFSv16.2.0 milestone Feb 25, 2022

This was referenced Mar 1, 2022

Convert modulefiles to LUA format (develop) #670

Closed

Introduce version files and their usage in develop branch #671

Closed

KateFriedman-NOAA closed this as completed Aug 3, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Convert modulefiles to lua and hpc-stack (v16) #639

Convert modulefiles to lua and hpc-stack (v16) #639

WalterKolczynski-NOAA commented Feb 8, 2022 •

edited by KateFriedman-NOAA

Loading

GeorgeGayno-NOAA commented Feb 8, 2022

KateFriedman-NOAA commented Feb 8, 2022

malloryprow commented Feb 9, 2022 •

edited

Loading

MichaelLueken commented Feb 15, 2022

KateFriedman-NOAA commented Feb 15, 2022

KateFriedman-NOAA commented Feb 15, 2022

MichaelLueken commented Feb 16, 2022

KateFriedman-NOAA commented Feb 16, 2022

MichaelLueken commented Feb 16, 2022

KateFriedman-NOAA commented Feb 16, 2022

MichaelLueken commented Feb 17, 2022

KateFriedman-NOAA commented Feb 17, 2022

MichaelLueken commented Feb 17, 2022

KateFriedman-NOAA commented Feb 17, 2022

KateFriedman-NOAA commented Feb 17, 2022

malloryprow commented Feb 18, 2022

KateFriedman-NOAA commented Feb 18, 2022

KateFriedman-NOAA commented Aug 3, 2022

Convert modulefiles to lua and hpc-stack (v16) #639

Convert modulefiles to lua and hpc-stack (v16) #639

Comments

WalterKolczynski-NOAA commented Feb 8, 2022 • edited by KateFriedman-NOAA Loading

GeorgeGayno-NOAA commented Feb 8, 2022

KateFriedman-NOAA commented Feb 8, 2022

malloryprow commented Feb 9, 2022 • edited Loading

MichaelLueken commented Feb 15, 2022

KateFriedman-NOAA commented Feb 15, 2022

KateFriedman-NOAA commented Feb 15, 2022

MichaelLueken commented Feb 16, 2022

KateFriedman-NOAA commented Feb 16, 2022

MichaelLueken commented Feb 16, 2022

KateFriedman-NOAA commented Feb 16, 2022

MichaelLueken commented Feb 17, 2022

KateFriedman-NOAA commented Feb 17, 2022

MichaelLueken commented Feb 17, 2022

KateFriedman-NOAA commented Feb 17, 2022

KateFriedman-NOAA commented Feb 17, 2022

malloryprow commented Feb 18, 2022

KateFriedman-NOAA commented Feb 18, 2022

KateFriedman-NOAA commented Aug 3, 2022

WalterKolczynski-NOAA commented Feb 8, 2022 •

edited by KateFriedman-NOAA

Loading

malloryprow commented Feb 9, 2022 •

edited

Loading