Create JEDI class

[DavidNew-NOAA]

Description

This PR creates a PyGFS class called JEDI, which is to be instantiated everytime a JEDI application is run. The AtmAnalysis and AtmEnsAnalysis classes are no longer children of the Analysis class, but rather direct children of the Task class. They each have a JEDI object as an attribute, which is used to run either the variational/ensemble DA JEDI applications or the FV3 increment converter JEDI application, depending on which job they are created for (e.g. atmanlvar vs. atmanlfv3inc). The intention is that a later PR will apply this framework to all analysis task, and the PyGFS Analysis class will be removed.

Type of change

• New feature (adds functionality)

Change characteristics

• Is this a breaking change (a change in existing functionality)? YES
• Does this change require a documentation update? NO

How has this been tested?

This is just a draft as of right now.

Checklist

• Any dependent changes have been merged and published
• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• My changes generate no new warnings
• New and existing tests pass with my changes
• I have made corresponding changes to the documentation if necessary

[CoryMartin-NOAA]

I don't have any major comments here, but I'm worried that we may be missing something that would make this more generic. @DavidNew-NOAA do you think it is reasonable to try this for another app (snow, aero, soca) at this stage to see if the JEDI class is generic enough?

[DavidNew-NOAA]

@CoryMartin-NOAA Sure, good idea. I haven't run any of those analyses in the past, but I'm willing to give them a try with the JEDI class.

[CoryMartin-NOAA]

@DavidNew-NOAA there are CI test cases for all of the above (I think). I would suggest starting with snow DA as it is the simplest, then aero. Let me know if you need help setting up an experiment using the CI test case(s)

[DavidNew-NOAA]

@guillaumevernieres Feel free to take a look at what I've done here. I'm still in draft mode.

Re: your comments on my previous (now closed) PR for context:

That PR was based on the idea of an inheritance family tree like so: Task->JEDI->AtmAnalysis/AtmEnsAnalysis/etc, but after some discussion with @aerorahul we are going to make AtmAnalysis/AtmEnsAnalysis/etc direct children of the Task class, and make the JEDI class an attribute of AtmAnalysis/AtmEnsAnalysis/etc. So every time you want to run a JEDI application in a task, you instantiate a JEDI object which contains all the information and methods necessary for initializing and running that application.

You mentioned that sometimes you want to run multiple JEDI applications in one set of tasks. It's a bit subtle, but in the case of AtmAnalysis, for example, that class only creates one JEDI object, but depending on whether I'm running the gdasatmanlvar job or the gdasatmanlfv3inc job, that JEDI object received a different JEDIEXE and JCB_BASE_YAML environment variable from config.atmanl+config.atmanlvar vs config.atmanlfv3inc, and thus they run a different JEDI executable with a different YAML depending on the job.

[guillaumevernieres]

Looks good @DavidNew-NOAA and it makes more sense if it's meant to be used to create data members in our tasks rather than inherit a rigid framework. That should replace part of a utility module we currently use for our b-matrix job.

I can see how to use your work to cleanup the marine-da/soca b-matrix task and run multiple jedi applications ... but we're so far behind on so many other things that it probably won't happen for a while!

[RussTreadon-NOAA]

@DavidNew-NOAA , I am about to begin work on g-w issue #2415 in RussTreadon-NOAA:feature/ensda_obs.

Do you recommend that I wait for this JEDI class PR to be merged into develop before adding a JEDI eobs equivalent to g-w?

[DavidNew-NOAA]

@RussTreadon-NOAA I still need Rahul's feedback, and he'a on leave, so I wouldn't wait.

[aerorahul]

I only reviewed jedi.py as that sets the foundation for this PR. I am open to your (and others) thoughts on my comments.

[emcbot]

CI Update on Wcoss2 at 09/06/24 03:12:11 PM
============================================
Cloning and Building global-workflow PR: 2805
with PID: 186518 on host: dlogin03

[emcbot]

Build FAILED on Orion in Build# 4 with error logs:

/work2/noaa/stmp/CI/ORION/2805/gfs/sorc/logs/build_upp.log

Follow link here to view the contents of the above file(s): (link)

[WalterKolczynski-NOAA]

Ignore Orion. Workflow in general doesn't work there yet.

[emcbot]

Automated global-workflow Testing Results:

Machine: Wcoss2
Start: Fri Sep  6 15:18:22 UTC 2024 on dlogin03
---------------------------------------------------
Build: Completed at 09/06/24 04:06:13 PM
Case setup: Completed for experiment C48_ATM_f445e28d
Case setup: Skipped for experiment C48mx500_3DVarAOWCDA_f445e28d
Case setup: Skipped for experiment C48_S2SWA_gefs_f445e28d
Case setup: Completed for experiment C48_S2SW_f445e28d
Case setup: Completed for experiment C96_atm3DVar_extended_f445e28d
Case setup: Skipped for experiment C96_atm3DVar_f445e28d
Case setup: Completed for experiment C96C48_hybatmaerosnowDA_f445e28d
Case setup: Completed for experiment C96C48_hybatmDA_f445e28d
Case setup: Completed for experiment C96C48_ufs_hybatmDA_f445e28d

[aerorahul]

Ignore Orion. Workflow in general doesn't work there yet.

The workflow does with the exception of cycling because the UPP submodule. It is being addressed in #2877

[emcbot]

CI Passed on Hercules in Build# 3
Built and ran in directory /work2/noaa/stmp/CI/HERCULES/2805

Experiment C48_ATM_f445e28d Completed 1 Cycles: *SUCCESS* at Fri Sep  6 12:18:24 CDT 2024
Experiment C48_S2SW_f445e28d Completed 1 Cycles: *SUCCESS* at Fri Sep  6 13:18:59 CDT 2024
Experiment C96_atm3DVar_f445e28d Completed 3 Cycles: *SUCCESS* at Fri Sep  6 13:25:15 CDT 2024
Experiment C96C48_hybatmDA_f445e28d Completed 3 Cycles: *SUCCESS* at Fri Sep  6 13:31:23 CDT 2024
Experiment C48_S2SWA_gefs_f445e28d Completed 1 Cycles: *SUCCESS* at Fri Sep  6 14:44:43 CDT 2024

[emcbot]

All CI Test Cases Passed on Wcoss2:

Experiment C48_ATM_f445e28d *** SUCCESS *** at 09/06/24 05:36:07 PM
Experiment C48_S2SW_f445e28d *** SUCCESS *** at 09/06/24 05:45:14 PM
Experiment C96C48_hybatmDA_f445e28d *** SUCCESS *** at 09/06/24 07:06:23 PM
Experiment C96C48_hybatmaerosnowDA_f445e28d *** SUCCESS *** at 09/06/24 07:36:21 PM
Experiment C96C48_ufs_hybatmDA_f445e28d *** SUCCESS *** at 09/06/24 08:57:15 PM
Experiment C96_atm3DVar_extended_f445e28d *** SUCCESS *** at 09/07/24 06:06:40 AM

[RussTreadon-NOAA]

Thank you @WalterKolczynski-NOAA!