Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Archive the experiment directory along with git status/diff output #3105

Merged
merged 24 commits into from
Dec 9, 2024
Merged

Archive the experiment directory along with git status/diff output #3105

merged 24 commits into from
Dec 9, 2024

Conversation

DavidHuber-NOAA
Copy link
Contributor

@DavidHuber-NOAA DavidHuber-NOAA commented Nov 15, 2024 •
edited
Loading

Description

This adds the capability to archive the experiment directory. Additionally, this adds options to run git status and git diff on the HOMEgfs global workflow (but not the submodules) and store that information within the experiment directory's archive. These options are specified in config.base with the following defaults:

export ARCH_EXPDIR='YES'     # Archive the EXPDIR configs, XML, and database
export ARCH_EXPDIR_FREQ=0    # How often to archive the EXPDIR in hours or 0 for first and last cycle only
export ARCH_HASHES='YES'     # Archive the hashes of the GW and submodules and 'git status' for each; requires ARCH_EXPDIR
export ARCH_DIFFS='NO'       # Archive the output of 'git diff' for the GW; requires ARCH_EXPDIR

Resolves #2994

Type of change

  • New feature (adds functionality)

Change characteristics

  • Is this a breaking change (a change in existing functionality)? NO
  • Does this change require a documentation update? YES
  • Does this change require an update to any of the following submodules? YES (If YES, please add a link to any PRs that are pending.)

How has this been tested?

  • Local archiving on Hercules for a C48_ATM case
  • Cycled testing on Hercules with ARCH_DIFFS=YES and ARCH_EXPDIR_FREQ=6,12
  • Testing with ARCH_EXPDIR=NO or ARCH_HASHES=NO

Checklist

  • Any dependent changes have been merged and published
  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have documented my code, including function, input, and output descriptions
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • This change is covered by an existing CI test or a new one has been added
  • Any new scripts have been added to the .github/CODEOWNERS file with owners
  • I have made corresponding changes to the system documentation if necessary

@DavidHuber-NOAA
Copy link
Contributor Author

FYI @AndrewEichmann-NOAA

@DavidHuber-NOAA DavidHuber-NOAA marked this pull request as ready for review November 18, 2024 16:23
@DavidHuber-NOAA
Copy link
Contributor Author

Tests were performed on Hercules and the experiment directory tarballs for both cycled and forecast-only modes were found to have been created as expected. Opening for review.

WalterKolczynski-NOAA
WalterKolczynski-NOAA requested changes Dec 3, 2024
View reviewed changes
parm/archive/expdir.yaml.j2 Outdated Show resolved Hide resolved
parm/archive/expdir.yaml.j2 Outdated Show resolved Hide resolved
parm/archive/master_gdas.yaml.j2
Comment on lines +101 to +106
# Archive the EXPDIR if requested
{% if archive_expdir %}
{% filter indent(width=4) %}
{% include "expdir.yaml.j2" %}
{% endfilter %}
{% endif %}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should only need this in one of gdas or gfs, but there is a bit of a coordination problem since either can be run without the other.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. I added a method to archive.py to determine which RUN to archive this in and set the archive_expdir boolean accordingly.

ush/python/pygfs/task/archive.py Outdated Show resolved Hide resolved
ush/python/pygfs/task/archive.py Outdated Show resolved Hide resolved
ush/python/pygfs/task/archive.py Outdated Show resolved Hide resolved
ush/python/pygfs/task/archive.py Outdated Show resolved Hide resolved
@DavidHuber-NOAA @WalterKolczynski-NOAA
Apply suggestions from code review
7ab8dc1 
Co-authored-by: Walter Kolczynski - NOAA <[email protected]>
@WalterKolczynski-NOAA WalterKolczynski-NOAA added the CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera label Dec 5, 2024
@emcbot emcbot added CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully and removed CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress labels Dec 5, 2024
@emcbot
Copy link

emcbot commented Dec 5, 2024

CI Passed on Hera in Build# 1
Built and ran in directory /scratch1/NCEPDEV/global/CI/3105


Experiment C48_ATM_729be32c Completed 2 Cycles: *SUCCESS* at Thu Dec  5 04:01:45 UTC 2024
Experiment C96_S2SWA_gefs_replay_ics_729be32c Completed 1 Cycles: *SUCCESS* at Thu Dec  5 04:26:50 UTC 2024
Experiment C48mx500_3DVarAOWCDA_729be32c Completed 2 Cycles: *SUCCESS* at Thu Dec  5 04:38:12 UTC 2024
Experiment C96C48_hybatmDA_729be32c Completed 3 Cycles: *SUCCESS* at Thu Dec  5 05:51:36 UTC 2024
Experiment C96C48_hybatmaerosnowDA_729be32c Completed 3 Cycles: *SUCCESS* at Thu Dec  5 05:51:37 UTC 2024
Experiment C96_atm3DVar_729be32c Completed 3 Cycles: *SUCCESS* at Thu Dec  5 05:59:14 UTC 2024
Experiment C48_S2SW_729be32c Completed 2 Cycles: *SUCCESS* at Thu Dec  5 06:22:39 UTC 2024
Experiment C96C48_ufs_hybatmDA_729be32c Completed 3 Cycles: *SUCCESS* at Thu Dec  5 06:52:24 UTC 2024
Experiment C48_S2SWA_gefs_729be32c Completed 1 Cycles: *SUCCESS* at Thu Dec  5 07:12:34 UTC 2024

@DavidHuber-NOAA DavidHuber-NOAA merged commit 3a8697d into NOAA-EMC:develop Dec 9, 2024
6 checks passed
@DavidHuber-NOAA DavidHuber-NOAA deleted the feature/archive_expdir branch December 9, 2024 14:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@WalterKolczynski-NOAA WalterKolczynski-NOAA WalterKolczynski-NOAA approved these changes

@aerorahul aerorahul Awaiting requested review from aerorahul

@AndrewEichmann-NOAA AndrewEichmann-NOAA Awaiting requested review from AndrewEichmann-NOAA

Assignees
No one assigned
Labels
CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Ability to save experiment directory (EXPDIR)
3 participants
@DavidHuber-NOAA @emcbot @WalterKolczynski-NOAA