JCB-based obs+bias staging, Jedi class updates, and marine B-matrix refactoring #2992
Conversation
…e bias correction files from tarball (NOAA-EMC#2862)
…on files using jedi class (NOAA-EMC#2862)
…ns analysis scripts (NOAA-EMC#2862)
emcbot
added
the
CI-Hera-Passed
|
CI Passed on Hera in Build# 2 |
|
Hercules C96C48_hybatmDA failure Examine log files Both forecast jobs died near the start of the model integration with nan messages This is an odd failure. C96C48_hybatmDA is GSI-based atmospheric DA. This PR does not alter either the forecast model or GSI-based DA. Install DavidNew-NOAA:feature/jcb-obsbias at 32534bb on Hercules in Examine output from Compare this with the successful rerun Comparison of the enkfstat files from the two runs show no differences apart from timing statistics, different order of printout, and ensemble mean analysis increments. I have not seen this error in my many runs of C96C48_hybatmDA on Hercules. A rerun of C96C48_hybatmDA via automated CI is likely to succeed. This, however, sidesteps the above failure. Why / how nonphysical increments were generated in this case remains unresolved. |
|
@WalterKolczynski-NOAA - What's the path forward from here? We would like to get this PR merged into
|
|
@aerorahul, @WalterKolczynski-NOAA , and @KateFriedman-NOAA I manually launched g-w DA CI on WCOSS2 (Dogwood). We know C96C48_hybatmaerosnowDA and C48mx500_3DVarAOWCDA will fail. GDASApp must be built with spack-stack on WCOSS2 in order for select jobs in these configurations to successfully run to complete. GSI and JEDI based atmospheric DA, C96C48_hybatmDA and C96C48_ufs_hybatmDA, should pass. The following g-w CI configuration do NOT contain a wcoss2 skip
Do these three configurations also need to be run on Dogwood? |
Unfortunately, the CI does not (yet) have the capability to select individual tests. We have to rerun them all on a platform. Orion is not needed. Orion's support level has been downgraded and most of the things that would cause a failure on Orion will also cause a failure on Hercules anyway.
Any tests that aren't being skipped should be run. It looks like the upgrade it postponed, so we should be able to get these in. I will run them overnight so hopefully they are complete by morning. |
WalterKolczynski-NOAA
added
CI-Wcoss2-Building
emcbot
added
CI-Hercules-Building
WalterKolczynski-NOAA
added
CI-Wcoss2-Building
|
CI Tests set up to run in /lfs/h2/emc/ptmp/walter.kolczynski/PR/PR_2992/RUNTESTS on WCOSS |
emcbot
added
CI-Hercules-Running
|
CI Passed on Hercules in Build# 3 |
|
WCOSS2 g-w CI
with the following results All jobs in all configurations except C96C48_hybatmaerosnowDA and C48mx500_3DVarAOWCDA successfully ran to completion. Note that C96C48_hybatmaerosnowDA and C48mx500_3DVarAOWCDA are set to be skipped on WCOSS2. If GDASApp is rebuilt and run with The following configurations in These ci configurations, apart from the three configurations which include JEDI based DA, were not run on Dogwood. Based on the above g-w CI results we can change the CI-Wcoss2-Running label to CI-Wcoss2-Passed. |
RussTreadon-NOAA
added
CI-Wcoss2-Passed
|
Acceptable (explainable) g-w CI results have been obtained on Hera, Hercules, and WCOSS2 (Dogwood). @WalterKolczynski-NOAA , @aerorahul , @KateFriedman-NOAA : Are any more tests required before this PR can be merged into |
WalterKolczynski-NOAA
merged commit 7ff942e
into
NOAA-EMC:develop
|
Thank you @WalterKolczynski-NOAA |
…efactoring (NOAA-EMC#2992) This PR is a companion to GDASApp PR NOAA-EMC/GDASApp#1312 (merged) and NOAA-EMC/jcb-gdas#31 (merged). This PR does three things: 1. It changes the observation and bias staging for the atmospheric analysis tasks to use JCB templates instead of reading the full JEDI input configuration dictionary in order to construct a list of files to stage. This is cleaner and places fewer constraints on how to initialize the analysis. 2. The ```Jedi``` constructor now takes as input a dictionary that is essentially subset of the ```task_config``` dictionary. This makes the code clearer and less opaque and makes debugging easier. Each dictionary is constructed from a YAML file with configuration parameters for each JEDI application that is run. 3. All JEDI applications and their input YAMLs are now initialized in the initialize job of the ```AtmAnalysis``` and ```AtmEnsAnalysis```. Before, in the ```atmensanl*``` jobs for example, the LETKF solver was initialized in the ```atmensanlinit```cjob, but the LETKF solver and FV3 increment converter were both initialized and executed in the ```atmensanlobs``` and ```atmensanlfv3inc``` jobs respectively. This makes more sense in terms of resource allocation. Addendum: I'm now rolling in the refactoring of the marine B-matrix task into this PR. That makes it also a companion of NOAA-EMC/GDASApp#1346 and NOAA-EMC/jcb-gdas#36. These new changes introduce the ```Jedi``` class and JCB into the marine B-matrix job. Partially resolvesNOAA-EMC/GDASApp#1296 --------- Co-authored-by: RussTreadon-NOAA <[email protected]> Co-authored-by: Rahul Mahajan <[email protected]>
Description
This PR is a companion to GDASApp PR #1312 (merged) and JCB-GDAS PR #31 (merged).
This PR does three things:
Jediconstructor now takes as input a dictionary that is essentially subset of thetask_configdictionary. This makes the code clearer and less opaque and makes debugging easier. Each dictionary is constructed from a YAML file with configuration parameters for each JEDI application that is run.AtmAnalysisandAtmEnsAnalysis. Before, in theatmensanl*jobs for example, the LETKF solver was initialized in theatmensanlinitcjob, but the LETKF solver and FV3 increment converter were both initialized and executed in theatmensanlobsandatmensanlfv3incjobs respectively. This makes more sense in terms of resource allocation.Addendum:
I'm now rolling in the refactoring of the marine B-matrix task into this PR. That makes it also a companion of GDASApp PR #1346 and JCB-GDAS PR #36.
These new changes introduce the
Jediclass and JCB into the marine B-matrix job.Partially resolves GDASApp issue #1296
Type of change
Change characteristics
How has this been tested?
C96C48_ufs_hybatmDA CI runs successfully
GDASApp jjob tests pass successfully
Checklist