Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use stochastic restart patterns on rerun #3077

Merged
merged 17 commits into from
Dec 5, 2024
Merged

Use stochastic restart patterns on rerun #3077

merged 17 commits into from
Dec 5, 2024

Conversation

WalterKolczynski-NOAA
Copy link
Contributor

@WalterKolczynski-NOAA WalterKolczynski-NOAA commented Nov 7, 2024 •
edited
Loading

Description

The stochastic pattern restart files were not being copied into the input directory when restarting the model after a segment/failure. These files are now copied in and the stochini flag set to .true. in the namelist on a rerun.

The files are NOT copied in for non-rerun warm starts.

Also removes the restriction that stochastic physics cannot be run on member 0, as this is desired down the line. Additional settings are added to the fcst and efcs configs to retain the current behavior.

A bug was also discovered and corrected during this work. The stage, forecast, and archive job all assumed that ca_data tile files are always present, but these are only created when cellular automata is on. Now ca_data files are handled if CA is on. To implement this change, the DO_CA setting had to be moved from the forecast configs to base so it is available to the stage_ic and archive jobs.

Resolves #2937

Type of change

  • Bug fix (fixes something broken)

Change characteristics

  • Is this a breaking change (a change in existing functionality)? NO
  • Does this change require a documentation update? NO
  • Does this change require an update to any of the following submodules? NO

How has this been tested?

Multi-segment test on Hercules

Checklist

  • Any dependent changes have been merged and published
  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have documented my code, including function, input, and output descriptions
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • This change is covered by an existing CI test or a new one has been added
  • Any new scripts have been added to the .github/CODEOWNERS file with owners
  • I have made corresponding changes to the system documentation if necessary

aerorahul
aerorahul previously approved these changes Nov 12, 2024
View reviewed changes
Copy link
Contributor

@aerorahul aerorahul left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good to me. No comments.
Please invite review from @pjpegion @NeilBarton-NOAA

@WalterKolczynski-NOAA WalterKolczynski-NOAA added the CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera label Nov 13, 2024
@emcbot emcbot added CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed and removed CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress labels Nov 13, 2024
@emcbot
Copy link

emcbot commented Nov 13, 2024

Experiment C96_atm3DVar FAILED on Hera in Build# 1 in
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/EXPDIR/C96_atm3DVar_d9209edc

@emcbot
Copy link

emcbot commented Nov 13, 2024

Experiment C48mx500_3DVarAOWCDA FAILED on Hera in Build# 1 with error logs:

/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_d9209edc/logs/2021032412/gdas_fcst_seg0.log

Follow link here to view the contents of the above file(s): (link)

@emcbot
Copy link

emcbot commented Nov 13, 2024

Experiment C48mx500_3DVarAOWCDA FAILED on Hera in Build# 1 in
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/EXPDIR/C48mx500_3DVarAOWCDA_d9209edc

@emcbot
Copy link

emcbot commented Nov 13, 2024

Experiment C96C48_ufs_hybatmDA FAILED on Hera in Build# 1 with error logs:

/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C96C48_ufs_hybatmDA_d9209edc/logs/2024022318/gdas_fcst_seg0.log

Follow link here to view the contents of the above file(s): (link)

@emcbot
Copy link

emcbot commented Nov 13, 2024

Experiment C96C48_hybatmDA FAILED on Hera in Build# 1 with error logs:

/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C96C48_hybatmDA_d9209edc/logs/2021122018/gdas_fcst_seg0.log

Follow link here to view the contents of the above file(s): (link)

@emcbot
Copy link

emcbot commented Nov 13, 2024

Experiment C96C48_ufs_hybatmDA FAILED on Hera in Build# 1 in
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/EXPDIR/C96C48_ufs_hybatmDA_d9209edc

@emcbot
Copy link

emcbot commented Nov 13, 2024

Experiment C96C48_hybatmaerosnowDA FAILED on Hera in Build# 1 with error logs:

/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C96C48_hybatmaerosnowDA_d9209edc/logs/2021122012/gdas_fcst_seg0.log

Follow link here to view the contents of the above file(s): (link)

@emcbot
Copy link

emcbot commented Nov 13, 2024

Experiment C96C48_hybatmDA FAILED on Hera in Build# 1 in
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/EXPDIR/C96C48_hybatmDA_d9209edc

@emcbot
Copy link

emcbot commented Nov 13, 2024

Experiment C96C48_hybatmaerosnowDA FAILED on Hera in Build# 1 in
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/EXPDIR/C96C48_hybatmaerosnowDA_d9209edc

@emcbot
Copy link

emcbot commented Nov 13, 2024

Experiment C48_S2SWA_gefs FAILED on Hera in Build# 1 with error logs:

/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C48_S2SWA_gefs_d9209edc/logs/2021032312/gefs_fcst_mem000_seg1.log
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C48_S2SWA_gefs_d9209edc/logs/2021032312/gefs_fcst_mem001_seg1.log
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C48_S2SWA_gefs_d9209edc/logs/2021032312/gefs_fcst_mem002_seg1.log

Follow link here to view the contents of the above file(s): (link)

@emcbot
Copy link

emcbot commented Nov 13, 2024

Experiment C48_S2SWA_gefs FAILED on Hera in Build# 1 in
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/EXPDIR/C48_S2SWA_gefs_d9209edc

@emcbot emcbot added CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed and removed CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed labels Nov 13, 2024
@emcbot
Copy link

emcbot commented Nov 13, 2024

CI Failed on Hera in Build# 1
Built and ran in directory /scratch1/NCEPDEV/global/CI/3077


Experiment C48mx500_3DVarAOWCDA_d9209edc Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Wed Nov 13 11:45:45 UTC 2024
Experiment C48mx500_3DVarAOWCDA_d9209edc Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_d9209edc/logs/2021032412/gdas_fcst_seg0.log
Experiment C96_atm3DVar_d9209edc Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Wed Nov 13 11:45:50 UTC 2024
Experiment C96_atm3DVar_d9209edc Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C96_atm3DVar_d9209edc/logs/2021122018/gdas_fcst_seg0.log
Experiment C96C48_hybatmDA_d9209edc Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Wed Nov 13 11:51:56 UTC 2024
Experiment C96C48_hybatmDA_d9209edc Terminated: *FAIL*
Experiment C96C48_ufs_hybatmDA_d9209edc Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Wed Nov 13 11:51:57 UTC 2024
Experiment C96C48_ufs_hybatmDA_d9209edc Terminated: *FAIL*
Experiment C96C48_hybatmaerosnowDA_d9209edc Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Wed Nov 13 11:51:59 UTC 2024
Experiment C96C48_hybatmaerosnowDA_d9209edc Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C96C48_hybatmDA_d9209edc/logs/2021122018/gdas_fcst_seg0.log
Error logs:
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C96C48_ufs_hybatmDA_d9209edc/logs/2024022318/gdas_fcst_seg0.log
Error logs:
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C96C48_hybatmaerosnowDA_d9209edc/logs/2021122012/gdas_fcst_seg0.log
Experiment C48_S2SWA_gefs_d9209edc Terminated with 0
FAIL
FAIL tasks failed and 3 dead at Wed Nov 13 15:37:41 UTC 2024
Experiment C48_S2SWA_gefs_d9209edc Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C48_S2SWA_gefs_d9209edc/logs/2021032312/gefs_fcst_mem000_seg1.log
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C48_S2SWA_gefs_d9209edc/logs/2021032312/gefs_fcst_mem001_seg1.log
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C48_S2SWA_gefs_d9209edc/logs/2021032312/gefs_fcst_mem002_seg1.log
Experiment C96_S2SWA_gefs_replay_ics_d9209edc Completed 1 Cycles: *SUCCESS* at Wed Nov 13 17:25:52 UTC 2024
Experiment C48_ATM_d9209edc Completed 2 Cycles: *SUCCESS* at Wed Nov 13 18:45:20 UTC 2024
Experiment C48_S2SW_d9209edc Completed 2 Cycles: *SUCCESS* at Wed Nov 13 23:13:39 UTC 2024

@emcbot emcbot removed the CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera label Dec 4, 2024
@WalterKolczynski-NOAA WalterKolczynski-NOAA added CI-Wcoss2-Passed **Bot use only** CI testing on WCOSS for this PR has completed successfully and removed CI-Wcoss2-Running **Bot use only** CI testing on WCOSS for this PR is in-progress labels Dec 4, 2024
@emcbot
Copy link

emcbot commented Dec 4, 2024

Experiment C96C48_hybatmaerosnowDA FAILED on Hera in Build# 5 with error logs:

/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C96C48_hybatmaerosnowDA_33615413/logs/2021122018/enkfgdas_earc01.log

Follow link here to view the contents of the above file(s): (link)

@emcbot emcbot added CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed and removed CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress labels Dec 4, 2024
@emcbot
Copy link

emcbot commented Dec 4, 2024

Experiment C96C48_hybatmaerosnowDA FAILED on Hera in Build# 5 in
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/EXPDIR/C96C48_hybatmaerosnowDA_33615413

@emcbot
Copy link

emcbot commented Dec 4, 2024

Experiment C48_S2SW FAILED on Hera in Build# 5 in
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/EXPDIR/C48_S2SW_33615413

@emcbot
Copy link

emcbot commented Dec 4, 2024

Experiment C96C48_hybatmDA FAILED on Hera in Build# 5 in
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/EXPDIR/C96C48_hybatmDA_33615413

@emcbot
Copy link

emcbot commented Dec 4, 2024

Experiment C96_atm3DVar FAILED on Hera in Build# 5 in
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/EXPDIR/C96_atm3DVar_33615413

@emcbot
Copy link

emcbot commented Dec 4, 2024

Experiment C96C48_ufs_hybatmDA FAILED on Hera in Build# 5 in
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/EXPDIR/C96C48_ufs_hybatmDA_33615413

@emcbot
Copy link

emcbot commented Dec 4, 2024

Experiment C48_S2SWA_gefs FAILED on Hera in Build# 5 in
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/EXPDIR/C48_S2SWA_gefs_33615413

@emcbot emcbot added CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed and removed CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed labels Dec 4, 2024
@emcbot
Copy link

emcbot commented Dec 4, 2024

CI Failed on Hera in Build# 5
Built and ran in directory /scratch1/NCEPDEV/global/CI/3077


Experiment C48mx500_3DVarAOWCDA_33615413 Completed 2 Cycles: *SUCCESS* at Wed Dec  4 20:52:51 UTC 2024
Experiment C48_ATM_33615413 Completed 2 Cycles: *SUCCESS* at Wed Dec  4 21:11:03 UTC 2024
Experiment C96_S2SWA_gefs_replay_ics_33615413 Completed 1 Cycles: *SUCCESS* at Wed Dec  4 21:17:37 UTC 2024
Experiment C96C48_hybatmaerosnowDA_33615413 Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Wed Dec  4 21:29:38 UTC 2024
Experiment C96C48_hybatmaerosnowDA_33615413 Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/3077/RUNTESTS/COMROOT/C96C48_hybatmaerosnowDA_33615413/logs/2021122018/enkfgdas_earc01.log

@WalterKolczynski-NOAA WalterKolczynski-NOAA added CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera and removed CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed labels Dec 4, 2024
@emcbot emcbot added CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully and removed CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress labels Dec 4, 2024
@emcbot
Copy link

emcbot commented Dec 5, 2024

CI Passed on Hera in Build# 6
Built and ran in directory /scratch1/NCEPDEV/global/CI/3077


Experiment C48_ATM_9ca245ef Completed 2 Cycles: *SUCCESS* at Thu Dec  5 00:51:54 UTC 2024
Experiment C96_S2SWA_gefs_replay_ics_9ca245ef Completed 1 Cycles: *SUCCESS* at Thu Dec  5 01:23:03 UTC 2024
Experiment C48_S2SW_9ca245ef Completed 2 Cycles: *SUCCESS* at Thu Dec  5 03:24:49 UTC 2024
Experiment C48_S2SWA_gefs_9ca245ef Completed 1 Cycles: *SUCCESS* at Thu Dec  5 03:56:47 UTC 2024
Experiment C48mx500_3DVarAOWCDA_9ca245ef Completed 2 Cycles: *SUCCESS* at Thu Dec  5 04:18:34 UTC 2024
Experiment C96C48_hybatmaerosnowDA_9ca245ef Completed 3 Cycles: *SUCCESS* at Thu Dec  5 05:38:22 UTC 2024
Experiment C96_atm3DVar_9ca245ef Completed 3 Cycles: *SUCCESS* at Thu Dec  5 05:44:20 UTC 2024
Experiment C96C48_hybatmDA_9ca245ef Completed 3 Cycles: *SUCCESS* at Thu Dec  5 05:44:26 UTC 2024
Experiment C96C48_ufs_hybatmDA_9ca245ef Completed 3 Cycles: *SUCCESS* at Thu Dec  5 06:45:05 UTC 2024

KateFriedman-NOAA
KateFriedman-NOAA approved these changes Dec 5, 2024
View reviewed changes
Copy link
Member

@KateFriedman-NOAA KateFriedman-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't comment on the science updates in this PR but the workflow changes look good. Thanks @WalterKolczynski-NOAA !

@WalterKolczynski-NOAA WalterKolczynski-NOAA merged commit bc61862 into NOAA-EMC:develop Dec 5, 2024
10 of 11 checks passed
@WalterKolczynski-NOAA WalterKolczynski-NOAA deleted the feature/stoch_restart branch December 5, 2024 16:04
This was referenced Dec 10, 2024
Open
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@CatherineThomas-NOAA CatherineThomas-NOAA CatherineThomas-NOAA left review comments

@bingfu-NOAA bingfu-NOAA bingfu-NOAA left review comments

@KateFriedman-NOAA KateFriedman-NOAA KateFriedman-NOAA approved these changes

@DavidHuber-NOAA DavidHuber-NOAA DavidHuber-NOAA left review comments

@aerorahul aerorahul aerorahul left review comments

@junwang-noaa junwang-noaa Awaiting requested review from junwang-noaa junwang-noaa is a code owner

@pjpegion pjpegion Awaiting requested review from pjpegion

@guillaumevernieres guillaumevernieres Awaiting requested review from guillaumevernieres

@AndrewEichmann-NOAA AndrewEichmann-NOAA Awaiting requested review from AndrewEichmann-NOAA

@DavidNew-NOAA DavidNew-NOAA Awaiting requested review from DavidNew-NOAA

@RussTreadon-NOAA RussTreadon-NOAA Awaiting requested review from RussTreadon-NOAA

@jiaruidong2017 jiaruidong2017 Awaiting requested review from jiaruidong2017

@CoryMartin-NOAA CoryMartin-NOAA Awaiting requested review from CoryMartin-NOAA

@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA Awaiting requested review from TerrenceMcGuinness-NOAA

@NeilBarton-NOAA NeilBarton-NOAA Awaiting requested review from NeilBarton-NOAA

Assignees
No one assigned
Labels
CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully CI-Hercules-Passed **Bot use only** CI testing on Hercules for this PR has completed successfully CI-Wcoss2-Passed **Bot use only** CI testing on WCOSS for this PR has completed successfully
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

stage atm_stoch.res.nc and ocn_stoch.res.nc during segment forecasts
8 participants
@WalterKolczynski-NOAA @emcbot @aerorahul @KateFriedman-NOAA @bingfu-NOAA @CatherineThomas-NOAA @DavidHuber-NOAA @NeilBarton-NOAA