Update global atmos upp job to use COMIN/COMOUT #2867

mingshichen-noaa · 2024-08-27T17:11:45Z

Description

NCO has requested that each COM variable specify whether it is an input or an output. This completes that process for the global-workflow Unified Post Processor (UPP) task.

Refs #2451

Type of change

Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO

How has this been tested?

Clone and build on RDHPCS
Cycled tests on Hercules
Forecast-only tests on Hercules

Checklist

Any dependent changes have been merged and published
My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
My changes generate no new warnings
New and existing tests pass with my changes
I have made corresponding changes to the documentation if necessary

WalterKolczynski-NOAA

Please be sure to run one of the cycled DA tests that will exercise this job (like C96C48_hybatmDA.yaml). Look for the gdasatmanlupp and gfsatmanlupp tasks.

jobs/JGLOBAL_ATMOS_UPP

ush/python/pygfs/task/upp.py

WalterKolczynski-NOAA · 2024-08-29T19:09:10Z

You need to undo the changes to the submodules.

When you pull in changes that update submodules, the submodules locally are not automatically updated. So when you do a git add, it commits the OLD versions back, which is definitely not what we want. Instead, you should have run git submodule update --init --recursive to update the submodules locally. Now, you need to check out the correct versions from develop and commit them (or rebase interactively to remove the offending commits, but that is a bit advanced).

aerorahul · 2024-09-03T14:30:24Z

jobs/JGLOBAL_ATMOS_UPP

+   COMIN_ATMOS_ANALYSIS:COM_ATMOS_ANALYSIS_TMPL \
+   COMIN_ATMOS_HISTORY:COM_ATMOS_HISTORY \
+   COMOUT_ATMOS_MASTER:COM_ATMOS_MASTER
+if [[ ! -d ${COMOUT_ATMOS_MASTER} ]]; then mkdir -m 775 -p "${COMOUT_ATMOS_MASTER}"; fi


Suggested change

if [[ ! -d ${COMOUT_ATMOS_MASTER} ]]; then mkdir -m 775 -p "${COMOUT_ATMOS_MASTER}"; fi

if [[ ! -d ${COMOUT_ATMOS_MASTER} ]]; then mkdir -p "${COMOUT_ATMOS_MASTER}"; fi

Please remove -m 775

I have removed "-m 775" on line 19.

@mingshichen-noaa
I still see -m 775 in this PR. Did you push your change?

@aerorahul
After I removed "-m 775" and then git add file, git commit -m ""-m 775" ,
finally git push origin HEAD:feature/atmos_upp_com_in_out.
Just now I tried to do the above mentioned final step again.
But after I checked jobs/JGLOBAL_ATMOS_UPP in the PR, "-m 775" is still over there. So far I have not identified the cause.

@aerorahul
I identified the reason why you see still "-m 775 in this PR". This is because I created two branches: origin/atmos_upp_com_in_out and origin/feature/atmos_upp_com_in_out. I did wrongly git push the modified files into origin/atmos_upp_com_in_out. in order to avoid the mistakes again, I have deleted the origin/atmos_upp_com_in_out.
Please review the PR again. Thanks.

aerorahul

just a note on -m 775 when creating the directory and revert the submodule pointers.
otherwise, looks good.

parm/post/upp.yaml

WalterKolczynski-NOAA · 2024-09-10T15:53:20Z

Also, even though it isn't showing as a conflict, wxflow needs to be reverted to the develop version the same way gdas and ufs_model were.

aerorahul

Looks good. Thanks.

emcbot · 2024-09-14T14:41:20Z

CI Update on Wcoss2 at 09/14/24 02:41:20 PM
============================================
Cloning and Building global-workflow PR: 2867
with PID: 106291 on host: dlogin03

emcbot · 2024-09-14T15:23:20Z

Automated global-workflow Testing Results:

Machine: Wcoss2
Start: Sat Sep 14 14:45:36 UTC 2024 on dlogin03
---------------------------------------------------
Build: Completed at 09/14/24 03:22:57 PM
Case setup: Completed for experiment C48_ATM_dddcaa81
Case setup: Skipped for experiment C48mx500_3DVarAOWCDA_dddcaa81
Case setup: Skipped for experiment C48_S2SWA_gefs_dddcaa81
Case setup: Completed for experiment C48_S2SW_dddcaa81
Case setup: Completed for experiment C96_atm3DVar_extended_dddcaa81
Case setup: Skipped for experiment C96_atm3DVar_dddcaa81
Case setup: Completed for experiment C96C48_hybatmaerosnowDA_dddcaa81
Case setup: Completed for experiment C96C48_hybatmDA_dddcaa81
Case setup: Completed for experiment C96C48_ufs_hybatmDA_dddcaa81

emcbot · 2024-09-15T04:45:10Z

All CI Test Cases Passed on Wcoss2:

Experiment C48_ATM_dddcaa81 *** SUCCESS *** at 09/14/24 04:57:13 PM
Experiment C48_S2SW_dddcaa81 *** SUCCESS *** at 09/14/24 05:06:14 PM
Experiment C96C48_hybatmDA_dddcaa81 *** SUCCESS *** at 09/14/24 06:03:27 PM
Experiment C96C48_hybatmaerosnowDA_dddcaa81 *** SUCCESS *** at 09/14/24 06:57:25 PM
Experiment C96C48_ufs_hybatmDA_dddcaa81 *** SUCCESS *** at 09/14/24 08:06:15 PM
Experiment C96_atm3DVar_extended_dddcaa81 *** SUCCESS *** at 09/15/24 04:42:34 AM

emcbot · 2024-09-16T06:12:53Z

CI Passed on Hercules in Build# 6
Built and ran in directory /work2/noaa/stmp/CI/HERCULES/2867


Experiment C48_ATM_dddcaa81 Completed 1 Cycles: *SUCCESS* at Sun Sep 15 23:10:58 CDT 2024
Experiment C96_atm3DVar_dddcaa81 Completed 3 Cycles: *SUCCESS* at Mon Sep 16 00:11:35 CDT 2024
Experiment C96C48_hybatmDA_dddcaa81 Completed 3 Cycles: *SUCCESS* at Mon Sep 16 00:17:48 CDT 2024
Experiment C48_S2SW_dddcaa81 Completed 1 Cycles: *SUCCESS* at Mon Sep 16 00:47:58 CDT 2024
Experiment C48_S2SWA_gefs_dddcaa81 Completed 1 Cycles: *SUCCESS* at Mon Sep 16 01:12:39 CDT 2024

mingshichen-noaa · 2024-09-16T14:01:48Z

@aerorahul @WalterKolczynski-NOAA
Thank you very much for your help on #2867 approved.

…al_obs_path * origin/develop: Update global atmos upp job to use COMIN/COMOUT (NOAA-EMC#2867) Update config.resources for bufr sounding job postsnd (NOAA-EMC#2917) Cleanup job for GEFS (NOAA-EMC#2919) Build GDASApp and unset memory in Gaea-C5 xml files (NOAA-EMC#2912) add 1 deg ocean/ice info to parm/config/gfs/config.resources (NOAA-EMC#2922) Support gefs C48 on Azure (NOAA-EMC#2881) Disable native grid writes for non-JEDI experiments; update C384 compression options (NOAA-EMC#2914)

NCO has requested that each COM variable specify whether it is an input or an output. This completes that process for the global-workflow Unified Post Processor (UPP) task. Refs: NOAA-EMC#2451

mingshichen-noaa and others added 8 commits August 12, 2024 09:53

changing COM_ into COMIN

91969c3

changing COM_ into COMIN_

f93185c

change COMIN_ATMOS_MASTER into COMOUT_ATMOS_MASTER

e4446c6

change COMIN_ATMOS_MASTER into COMOUT_ATMOS_MASTER

d842dd8

changing COM/ into COMIN/ or COMOUT/

f296840

update COM_ into COMIN_ or COMOUT_

ba61fad

Merge branch 'NOAA-EMC:develop' into feature/atmos_upp_com_in_out

0be05d7

Merge branch 'NOAA-EMC:develop' into feature/atmos_upp_com_in_out

435bdd7

mingshichen-noaa requested review from aerorahul, WalterKolczynski-NOAA and DavidHuber-NOAA August 27, 2024 17:13

WalterKolczynski-NOAA requested changes Aug 28, 2024

View reviewed changes

jobs/JGLOBAL_ATMOS_UPP Outdated Show resolved Hide resolved

ush/python/pygfs/task/upp.py Outdated Show resolved Hide resolved

ush/python/pygfs/task/upp.py Outdated Show resolved Hide resolved

mingshichen-noaa added 5 commits August 28, 2024 14:48

change COM to COMIN or COMOUT with explicitly specifying the template

668cfbd

gdas.cd and ufs_model.fd have been updated

4b08226

remove COMIN, only keep COMOUT

06ed674

modified indent space

70d6e15

update gdas.cd wxflow

09871b2

aerorahul reviewed Sep 3, 2024

View reviewed changes

mingshichen-noaa added 2 commits September 10, 2024 09:59

recover the add gdas.cd ufs_model.fd

5c74d5c

resolved file conflict of upp.yaml

0aeef60

WalterKolczynski-NOAA reviewed Sep 10, 2024

View reviewed changes

parm/post/upp.yaml Outdated Show resolved Hide resolved

mingshichen-noaa added 3 commits September 10, 2024 11:38

recover wxflow

99536e1

removed -m 755

f2c099f

resolve conflicts of upp.yaml

c459290

aerorahul previously approved these changes Sep 11, 2024

View reviewed changes

WalterKolczynski-NOAA approved these changes Sep 11, 2024

View reviewed changes

WalterKolczynski-NOAA added the CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera label Sep 11, 2024

aerorahul dismissed WalterKolczynski-NOAA’s stale review via 90f67ab September 14, 2024 14:39

aerorahul added CI-Wcoss2-Ready **CM use only** PR is ready for CI testing on WCOSS and removed CI-Wcoss2-Failed **Bot use only** CI testing on WCOSS for this PR has failed labels Sep 14, 2024

Merge branch 'develop' into feature/atmos_upp_com_in_out

dddcaa8

emcbot added CI-Wcoss2-Building **Bot use only** CI testing is cloning/building on WCOSS and removed CI-Wcoss2-Ready **CM use only** PR is ready for CI testing on WCOSS labels Sep 14, 2024

emcbot added CI-Wcoss2-Running **Bot use only** CI testing on WCOSS for this PR is in-progress and removed CI-Wcoss2-Building **Bot use only** CI testing is cloning/building on WCOSS labels Sep 14, 2024

emcbot added CI-Wcoss2-Passed **Bot use only** CI testing on WCOSS for this PR has completed successfully and removed CI-Wcoss2-Running **Bot use only** CI testing on WCOSS for this PR is in-progress labels Sep 15, 2024

WalterKolczynski-NOAA added CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules and removed CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed labels Sep 16, 2024

Merge branch 'develop' into feature/atmos_upp_com_in_out

970998c

aerorahul requested review from aerorahul and WalterKolczynski-NOAA September 16, 2024 13:56

aerorahul approved these changes Sep 16, 2024

View reviewed changes

aerorahul merged commit 2602eac into NOAA-EMC:develop Sep 16, 2024
5 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update global atmos upp job to use COMIN/COMOUT #2867

Update global atmos upp job to use COMIN/COMOUT #2867

mingshichen-noaa commented Aug 27, 2024

WalterKolczynski-NOAA left a comment

WalterKolczynski-NOAA commented Aug 29, 2024

aerorahul Sep 3, 2024

mingshichen-noaa Sep 3, 2024

aerorahul Sep 11, 2024

mingshichen-noaa Sep 11, 2024

mingshichen-noaa Sep 12, 2024

aerorahul left a comment

WalterKolczynski-NOAA commented Sep 10, 2024

aerorahul left a comment

emcbot commented Sep 14, 2024

emcbot commented Sep 14, 2024

emcbot commented Sep 15, 2024

emcbot commented Sep 16, 2024

mingshichen-noaa commented Sep 16, 2024

	if [[ ! -d ${COMOUT_ATMOS_MASTER} ]]; then mkdir -m 775 -p "${COMOUT_ATMOS_MASTER}"; fi
	if [[ ! -d ${COMOUT_ATMOS_MASTER} ]]; then mkdir -p "${COMOUT_ATMOS_MASTER}"; fi

Update global atmos upp job to use COMIN/COMOUT #2867

Update global atmos upp job to use COMIN/COMOUT #2867

Conversation

mingshichen-noaa commented Aug 27, 2024

Description

Type of change

Change characteristics

How has this been tested?

Checklist

WalterKolczynski-NOAA left a comment

Choose a reason for hiding this comment

WalterKolczynski-NOAA commented Aug 29, 2024

aerorahul Sep 3, 2024

Choose a reason for hiding this comment

mingshichen-noaa Sep 3, 2024

Choose a reason for hiding this comment

aerorahul Sep 11, 2024

Choose a reason for hiding this comment

mingshichen-noaa Sep 11, 2024

Choose a reason for hiding this comment

mingshichen-noaa Sep 12, 2024

Choose a reason for hiding this comment

aerorahul left a comment

Choose a reason for hiding this comment

WalterKolczynski-NOAA commented Sep 10, 2024

aerorahul left a comment

Choose a reason for hiding this comment

emcbot commented Sep 14, 2024

emcbot commented Sep 14, 2024

emcbot commented Sep 15, 2024

emcbot commented Sep 16, 2024

mingshichen-noaa commented Sep 16, 2024