UFS-dev PR#226

[grantfirl]

This is only a ccpp-physics submodule update for NCAR/ccpp-physics#1097

RT baseline changes are consistent with changes to ccpp physics changes.

[grantfirl]

RT results:

Output for arm_sgp_summer_1997_A_SCM_GFS_v17_p8_ugwpv1 is IDENTICAL to baseline
Output for arm_sgp_summer_1997_A_SCM_GFS_v16_RRTMGP is IDENTICAL to baseline
Output for arm_sgp_summer_1997_A_SCM_GFS_v16 is IDENTICAL to baseline
Output for arm_sgp_summer_1997_A_SCM_WoFS_v0 DIFFERS from baseline.
Output for arm_sgp_summer_1997_A_SCM_HRRR_gf DIFFERS from baseline.
Output for twpice_SCM_GFS_v17_p8_ugwpv1 is IDENTICAL to baseline
Output for twpice_SCM_GFS_v16_RRTMGP is IDENTICAL to baseline
Output for twpice_SCM_GFS_v16 is IDENTICAL to baseline
Output for twpice_SCM_WoFS_v0 DIFFERS from baseline.
Output for twpice_SCM_HRRR_gf DIFFERS from baseline.
Output for bomex_SCM_GFS_v17_p8_ugwpv1 is IDENTICAL to baseline
Output for bomex_SCM_GFS_v16_RRTMGP is IDENTICAL to baseline
Output for bomex_SCM_GFS_v16 is IDENTICAL to baseline
Output for bomex_SCM_WoFS_v0 is IDENTICAL to baseline
Output for bomex_SCM_HRRR_gf DIFFERS from baseline.
Output for astex_SCM_GFS_v17_p8_ugwpv1 is IDENTICAL to baseline
Output for astex_SCM_GFS_v16_RRTMGP is IDENTICAL to baseline
Output for astex_SCM_GFS_v16 is IDENTICAL to baseline
Output for astex_SCM_WoFS_v0 is IDENTICAL to baseline
Output for astex_SCM_HRRR_gf is IDENTICAL to baseline
Output for LASSO_2016051812_SCM_GFS_v17_p8_ugwpv1 is IDENTICAL to baseline
Output for LASSO_2016051812_SCM_GFS_v16_RRTMGP is IDENTICAL to baseline
Output for LASSO_2016051812_SCM_GFS_v16 is IDENTICAL to baseline
Output for LASSO_2016051812_SCM_WoFS_v0 is IDENTICAL to baseline
Output for LASSO_2016051812_SCM_HRRR_gf is IDENTICAL to baseline
Output for COMBLE_SCM_GFS_v17_p8_ugwpv1 is IDENTICAL to baseline
Output for COMBLE_SCM_GFS_v16_RRTMGP is IDENTICAL to baseline
Output for COMBLE_SCM_GFS_v16 is IDENTICAL to baseline
Output for COMBLE_SCM_WoFS_v0 DIFFERS from baseline.
Output for COMBLE_SCM_HRRR_gf DIFFERS from baseline.
Output for MOSAiC-AMPS_SCM_GFS_v17_p8_ugwpv1 is IDENTICAL to baseline
Output for MOSAiC-AMPS_SCM_GFS_v16_RRTMGP is IDENTICAL to baseline
Output for MOSAiC-AMPS_SCM_GFS_v16 is IDENTICAL to baseline
Output for MOSAiC-AMPS_SCM_WoFS_v0 DIFFERS from baseline.
Output for MOSAiC-AMPS_SCM_HRRR_gf is IDENTICAL to baseline
Output for gabls3_SCM_GFS_v16 is IDENTICAL to baseline
Output for arm_sgp_summer_1997_A_SCM_GFS_v17_p8 is IDENTICAL to baseline
Output for arm_sgp_summer_1997_A_SCM_HRRR DIFFERS from baseline.
Output for arm_sgp_summer_1997_A_SCM_RRFS_v1beta DIFFERS from baseline.
Output for arm_sgp_summer_1997_A_SCM_RAP DIFFERS from baseline.
Output for arm_sgp_summer_1997_A_SCM_GFS_v15p2 is IDENTICAL to baseline
Output for twpice_SCM_GFS_v17_p8 is IDENTICAL to baseline
Output for twpice_SCM_HRRR DIFFERS from baseline.
Output for twpice_SCM_RRFS_v1beta DIFFERS from baseline.
Output for twpice_SCM_RAP DIFFERS from baseline.
Output for twpice_SCM_GFS_v15p2 is IDENTICAL to baseline
Output for bomex_SCM_GFS_v17_p8 is IDENTICAL to baseline
Output for bomex_SCM_HRRR is IDENTICAL to baseline
Output for bomex_SCM_RRFS_v1beta is IDENTICAL to baseline
Output for bomex_SCM_RAP DIFFERS from baseline.
Output for bomex_SCM_GFS_v15p2 is IDENTICAL to baseline
Output for bomex_SCM_GFS_v16_debug is IDENTICAL to baseline
Output for astex_SCM_GFS_v17_p8 is IDENTICAL to baseline
Output for astex_SCM_HRRR is IDENTICAL to baseline
Output for astex_SCM_RRFS_v1beta is IDENTICAL to baseline
Output for astex_SCM_RAP is IDENTICAL to baseline
Output for astex_SCM_GFS_v15p2 is IDENTICAL to baseline
Output for LASSO_2016051812_SCM_GFS_v17_p8 is IDENTICAL to baseline
Output for LASSO_2016051812_SCM_HRRR is IDENTICAL to baseline
Output for LASSO_2016051812_SCM_RRFS_v1beta is IDENTICAL to baseline
Output for LASSO_2016051812_SCM_RAP is IDENTICAL to baseline
Output for LASSO_2016051812_SCM_GFS_v15p2 is IDENTICAL to baseline
ALL TESTS PASSED, BUT OUTPUT DIFFERS FROM BASELINE.

[dustinswales]

@grantfirl @scrasmussen @mkavulich It seems that the CI tests are now failing when the baseline comparison fails, and consequentially not creating new github artifacts that need to be used to get the tests passing again?
The CI tests shouldn't fail when baselines disagree?

Update What is this? https://github.com/NCAR/ccpp-scm/actions/runs/11411899398/job/31756974040?pr=533#step:24:1664

[grantfirl]

@grantfirl @scrasmussen @mkavulich It seems that the CI tests are now failing when the baseline comparison fails, and consequentially not creating new github artifacts that need to be used to get the tests passing again? The CI tests shouldn't fail when baselines disagree?

Update What is this? https://github.com/NCAR/ccpp-scm/actions/runs/11411899398/job/31756974040?pr=533#step:24:1664

I added a new test that runs schemes in the GFS_debug file for Dom. Let's concentrate on PRs in order and debug this when we get to it.

[mkavulich]

@dustinswales I don't think that's what's keeping the artifacts from being created, I think it's the new failing "GFS_debug" test. You can see that the single-precision artifacts were created. I do see the same message you do ('No artifacts will be uploaded.'). I honestly don't know what that's about, because they clearly are being uploaded. You can see the same message for all tests in the last PR here: https://github.com/NCAR/ccpp-scm/actions/runs/11446706947

[dustinswales]

@mkavulich Yeah, I didn't realize there was a new test being added in an earlier PR.
The artifact isn't being created because the RTs disagree with the baseline, but rather they are failing because the baseline does not exist.

[grantfirl]

@dustinswales @scrasmussen @mkavulich This is ready to review. Several RTs that use GF convection and MYNN PBL change results. Therefore, we'll need to upload the artifacts from this.

[grantfirl]

@mkavulich @scrasmussen I don't think that the RT artifacts from this PR were ever uploaded. The lack of updating baselines is causing unnecessary CI RT failures.

[scrasmussen]

RT artifacts

Hi @grantfirl, thanks for pointing this out. I'll start looking into this!

[scrasmussen]

@mkavulich @grantfirl I have access to the cluster to add these files now but am unsure of the process because I've never added the RT artifacts before.

Are the steps:

1. when merging PR#XYZ you also go to PR#XYZ's CI test to build and run SCM regression tests section to download the 5 artifacts that are generated
2. if the tests are passing, copy them over to the mohawk server?

Bonus question: should I add PR#536 RT artifacts instead of PR#533 because it is the most recently merged PR?

[grantfirl]

@mkavulich @grantfirl I have access to the cluster to add these files now but am unsure of the process because I've never added the RT artifacts before.

Are the steps:

1. when merging PR#XYZ you also go to PR#XYZ's CI test to build and run SCM regression tests section to download the 5 artifacts that are generated
2. if the tests are passing, copy them over to the mohawk server?

Bonus question: should I add PR#536 RT artifacts instead of PR#533 because it is the most recently merged PR?

@scrasmussen @mkavulich

Step 1 - yes
Step 2 - If tests are passing, then the baselines aren't changing. We want to make sure to update the baselines when everything runs as expected, but the RT results change (also as expected). This is something that should be mentioned in the PR comments or we should make a 'baseline change' tag like they did in ufs-weather-model.

Bonus - Yes, we should add baselines from the latest PR to get merged rather than this one.

[mkavulich]

@scrasmussen Yes that is roughtly the procedure I have been using: once a PR is merged, if there were baseline changes, I would go to the PR page (for example, the latest one merged is #536) click the "checks" tab, then scroll down to "CI test to build and run SCM regression tests". From there I would download the runtime artifacts at the bottom of the page to my local machine. Once I downloaded the baselines locally, I would then upload them to mohawk at the location /d2/www/dtcenter/ccpp/rt.

Theoretically you should be able to move them directly from the web to mohawk using wget or curl (skipping the intermediate local download), but I have never tried this.

[scrasmussen]

@mkavulich @grantfirl Thanks for the feedback! Working on getting the correct permissions to copy the files into the rt directory. Will let folks know when the files are successfully copied over.