Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Move machine-based options from config.base to host files #3053

Merged
merged 19 commits into from
Nov 13, 2024
Merged

Move machine-based options from config.base to host files #3053

merged 19 commits into from
Nov 13, 2024

Conversation

DavidHuber-NOAA
Copy link
Contributor

@DavidHuber-NOAA DavidHuber-NOAA commented Nov 1, 2024 •
edited
Loading

Description

This moves all machine-specific options to the workflow/hosts files from the config.* files.

This also turns HPSS archiving on for WCOSS2 by default.

Resolves #2942
Resolves #3087

Type of change

  • Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

  • Is this a breaking change (a change in existing functionality)? NO
  • Does this change require a documentation update? NO
  • Does this change require an update to any of the following submodules? NO

How has this been tested?

CI testing on Hercules. The tracker and genesis jobs did not run (as expected).
WCOSS2 testing still needs to be performed.

Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have documented my code, including function, input, and output descriptions
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • This change is covered by an existing CI test or a new one has been added

github-advanced-security[bot]
github-advanced-security bot found potential problems Nov 1, 2024
View reviewed changes
parm/config/gfs/config.prepoceanobs Fixed Show fixed Hide fixed
parm/config/gfs/config.prepoceanobs Fixed Show fixed Hide fixed
@DavidHuber-NOAA
Copy link
Contributor Author

This will break the C96_atm3DVar_extended test case until a fix is made for the gfs_downstream.tar file (as mentioned in #3019).

@DavidHuber-NOAA
Copy link
Contributor Author

There is an issue with the gfs_bufrsnd job that is preventing it from creating the output gfs_collective BUFR files. I am re-disabling HPSS archiving on WCOSS2 for the time being and will add this information to #3019.

Marking ready for review.

@DavidHuber-NOAA DavidHuber-NOAA marked this pull request as ready for review November 1, 2024 14:32
@DavidHuber-NOAA DavidHuber-NOAA added CI-Wcoss2-Building **Bot use only** CI testing is cloning/building on WCOSS CI-Wcoss2-Running **Bot use only** CI testing on WCOSS for this PR is in-progress CI-Wcoss2-Failed **Bot use only** CI testing on WCOSS for this PR has failed and removed CI-Wcoss2-Building **Bot use only** CI testing is cloning/building on WCOSS CI-Wcoss2-Running **Bot use only** CI testing on WCOSS for this PR is in-progress labels Nov 1, 2024
@DavidHuber-NOAA DavidHuber-NOAA removed the CI-Wcoss2-Failed **Bot use only** CI testing on WCOSS for this PR has failed label Nov 4, 2024
TerrenceMcGuinness-NOAA
TerrenceMcGuinness-NOAA previously approved these changes Nov 5, 2024
View reviewed changes
Copy link
Collaborator

@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a commendable PR and a key demonstration of how to refactor an "if-y diff-y" system specific configuration construct into a more centralized and pythonic approach.  I particularly liked how you enhanced the formation of the defaulting overrides.

@DavidHuber-NOAA DavidHuber-NOAA added the CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules label Nov 5, 2024
@emcbot emcbot added CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress and removed CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules labels Nov 5, 2024
WalterKolczynski-NOAA
WalterKolczynski-NOAA requested changes Nov 5, 2024
View reviewed changes
Copy link
Contributor

@WalterKolczynski-NOAA WalterKolczynski-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor things and one thing to sneak in (or leave for another time).

workflow/hosts/wcoss2.yaml Outdated Show resolved Hide resolved
parm/config/gefs/config.base Outdated Show resolved Hide resolved
workflow/setup_expt.py Outdated Show resolved Hide resolved
workflow/setup_expt.py Show resolved Hide resolved
@WalterKolczynski-NOAA WalterKolczynski-NOAA added CI-Wcoss2-Building **Bot use only** CI testing is cloning/building on WCOSS CI-Wcoss2-Running **Bot use only** CI testing on WCOSS for this PR is in-progress and removed CI-Wcoss2-Failed **Bot use only** CI testing on WCOSS for this PR has failed CI-Wcoss2-Building **Bot use only** CI testing is cloning/building on WCOSS labels Nov 13, 2024
@WalterKolczynski-NOAA
Copy link
Contributor

CI Tests set up to run in /lfs/h2/emc/ptmp/walter.kolczynski/PR/PR_3053/RUNTESTS on WCOSS

@WalterKolczynski-NOAA WalterKolczynski-NOAA added CI-Wcoss2-Building **Bot use only** CI testing is cloning/building on WCOSS CI-Wcoss2-Failed **Bot use only** CI testing on WCOSS for this PR has failed and removed CI-Wcoss2-Building **Bot use only** CI testing is cloning/building on WCOSS labels Nov 13, 2024
@WalterKolczynski-NOAA
Copy link
Contributor

CI Tests set up to run in /lfs/h2/emc/ptmp/walter.kolczynski/PR/PR_3053/RUNTESTS on WCOSS

@WalterKolczynski-NOAA WalterKolczynski-NOAA removed the CI-Wcoss2-Running **Bot use only** CI testing on WCOSS for this PR is in-progress label Nov 13, 2024
@WalterKolczynski-NOAA
Copy link
Contributor

C96C48_ufs_hybatmDA is failing because the gdas build fails due to issues with oops. I believe this is a known issue, but the test should be disabled in the meantime.

Other active tests pass.

@DavidHuber-NOAA
Copy link
Contributor Author

I disabled the UFS DA tests for WCOSS2 as part of this PR. Thanks for the heads up.

@aerorahul
Copy link
Contributor

@DavidHuber-NOAA
Do you have confidence that merging this is OK. You have run offline tests on WCOSS2, correct?

@DavidHuber-NOAA
Copy link
Contributor Author

Yes, I am confident in this PR. Here is a summary of testing done on WCOSS2:

  • I ran the setup scripts on WCOSS2 yesterday, and just now, to confirm the correct Python version is being used and that the aerosol input data paths are being set correctly.
  • I successfully ran CI tests on November 1st-2nd.
  • Walter ran CI tests yesterday (11/12). All tests passed except the UFS DA test, as (temporarily) expected.

@aerorahul
Copy link
Contributor

@RussTreadon-NOAA
If you are in agreement that disabling ufs DA tests are acceptable, please let us know. We would like to merge this PR.

@WalterKolczynski-NOAA WalterKolczynski-NOAA added CI-Wcoss2-Passed **Bot use only** CI testing on WCOSS for this PR has completed successfully and removed CI-Wcoss2-Failed **Bot use only** CI testing on WCOSS for this PR has failed labels Nov 13, 2024
@aerorahul aerorahul merged commit 3e78e58 into NOAA-EMC:develop Nov 13, 2024
5 checks passed
@aerorahul aerorahul mentioned this pull request Nov 13, 2024
Closed
WalterKolczynski-NOAA added a commit to WalterKolczynski-NOAA/global-workflow that referenced this pull request Nov 14, 2024
@WalterKolczynski-NOAA
Add mixing defaults entry for aero in GEFS
d953c49 
The defulats file for GEFS was missing an entry for aero to ensure
@field@ get parsed at expt setup time. This was added to gfs as
part of NOAA-EMC#3053, but gefs was missed.
@DavidHuber-NOAA DavidHuber-NOAA deleted the feature/machine_opts branch November 14, 2024 15:22
WalterKolczynski-NOAA added a commit to WalterKolczynski-NOAA/global-workflow that referenced this pull request Nov 22, 2024
@DavidHuber-NOAA @WalterKolczynski-NOAA @aerorahul
Move machine-based options from config.base to host files (NOAA-EMC#3053
56fd7f6 
)

This moves all machine-specific options to the workflow/hosts files from
the config.* files.

This also turns HPSS archiving on for WCOSS2 by default.
This turns OFF UFS DA tests on WCOSS2.

Resolves NOAA-EMC#2942
Resolves NOAA-EMC#3087

---------

Co-authored-by: Walter Kolczynski - NOAA <[email protected]>
Co-authored-by: Rahul Mahajan <[email protected]>
@CoryMartin-NOAA
Copy link
Contributor

I have a hunch (but no evidence) that #3124 may be related to this?

@WalterKolczynski-NOAA WalterKolczynski-NOAA mentioned this pull request Nov 25, 2024
Open
16 tasks
@DavidHuber-NOAA DavidHuber-NOAA mentioned this pull request Nov 26, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@WalterKolczynski-NOAA WalterKolczynski-NOAA WalterKolczynski-NOAA approved these changes

@aerorahul aerorahul aerorahul approved these changes

@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA Awaiting requested review from TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA is a code owner

Assignees
No one assigned
Labels
CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully CI-Hercules-Passed **Bot use only** CI testing on Hercules for this PR has completed successfully CI-Wcoss2-Passed **Bot use only** CI testing on WCOSS for this PR has completed successfully
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Jinja error during experiment creation on WCOSS Move all machine-specific options to workflow/hosts files
6 participants
@DavidHuber-NOAA @emcbot @aerorahul @WalterKolczynski-NOAA @CoryMartin-NOAA @TerrenceMcGuinness-NOAA