Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update snow obs preprocessing job #2946

Closed

Conversation

jiaruidong2017
Copy link
Contributor

Description

This PR modifies the workflow XML configuration so that the prep snow obs job only runs on the 00z cycle.

This PR also renames the snow_obs and snowobs to the snowcover for highlighting the input IMS data source as snow cover.

Resolves (#2902)

Type of change

  • Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

  • Is this a breaking change (a change in existing functionality)? NO
  • Does this change require a documentation update? NO
  • Does this change require an update to any of the following submodules? NO

How has this been tested?

  • Forecast-only on Hera

Checklist

  • Any dependent changes have been merged and published
  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • I have made corresponding changes to the documentation if necessary

Sorry, something went wrong.

if self._app_config.do_jedisnowda:
sdate_snocvr = self._base['SDATE']
edate_snocvr = self._base['EDATE']
interval_snocvr = to_timedelta(f"24:00:00H")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
interval_snocvr = to_timedelta(f"24:00:00H")
interval_snocvr = to_timedelta('24H')

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Made changes. Thanks.

task_dict = {'task_name': task_name,
'resources': resources,
'dependency': dependencies,
'envars': self.envars,
'cycledef': self.run.replace('enkf', ''),
'command': f'{self.HOMEgfs}/jobs/rocoto/prepsnowobs.sh',
'cycledef': cycledef,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
'cycledef': cycledef,
'cycledef': 'gdas_prep_snocvr',

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Made changes as suggested. Thanks.


deps = []
dep_dict = {'type': 'task', 'name': f'{self.run}prep'}
deps.append(rocoto.add_dependency(dep_dict))
dependencies = rocoto.create_dependency(dep=deps)

resources = self.get_resource('prepsnowobs')
task_name = f'{self.run}prepsnowobs'
cycledef = 'gdas_prep_snocvr'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
cycledef = 'gdas_prep_snocvr'

workflow/rocoto/gfs_tasks.py Outdated Show resolved Hide resolved
@WalterKolczynski-NOAA WalterKolczynski-NOAA added the blocked Issue is currently being blocked by another issue label Oct 1, 2024
@CoryMartin-NOAA
Copy link
Contributor

@jiaruidong2017 can you resolve conflicts and then, if all his comments are addressed, re-request review from @WalterKolczynski-NOAA ?

@jiaruidong2017
Copy link
Contributor Author

@WalterKolczynski-NOAA The <taskvalid> is used to update the task dependency as below.

image

I conducted the test run, and the run was successful. Would you please review this PR? Thanks.

@WalterKolczynski-NOAA WalterKolczynski-NOAA removed the blocked Issue is currently being blocked by another issue label Oct 23, 2024
@emcbot emcbot added CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed and removed CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed labels Oct 28, 2024
@emcbot
Copy link

emcbot commented Oct 28, 2024

CI Failed on Hera in Build# 5
Built and ran in directory /scratch1/NCEPDEV/global/CI/2946


Experiment C96_S2SWA_gefs_replay_ics_a00f9e8a Completed 1 Cycles: *SUCCESS* at Mon Oct 28 14:18:35 UTC 2024
Experiment C48mx500_3DVarAOWCDA_a00f9e8a Completed 2 Cycles: *SUCCESS* at Mon Oct 28 14:24:38 UTC 2024
Experiment C48_ATM_a00f9e8a Completed 2 Cycles: *SUCCESS* at Mon Oct 28 14:24:38 UTC 2024
Experiment C48_S2SWA_gefs_a00f9e8a Completed 1 Cycles: *SUCCESS* at Mon Oct 28 16:16:13 UTC 2024
Experiment C96_atm3DVar_a00f9e8a Completed 3 Cycles: *SUCCESS* at Mon Oct 28 16:21:17 UTC 2024
Experiment C96C48_hybatmDA_a00f9e8a Completed 3 Cycles: *SUCCESS* at Mon Oct 28 16:21:17 UTC 2024
Experiment C96C48_hybatmaerosnowDA_a00f9e8a Terminated with 0 tasks failed and 0 dead at Mon Oct 28 16:21:20 UTC 2024
Experiment C96C48_hybatmaerosnowDA_a00f9e8a Terminated: *STALLED*
Experiment C48_S2SW_a00f9e8a Completed 2 Cycles: *SUCCESS* at Mon Oct 28 16:27:30 UTC 2024
Experiment C96C48_ufs_hybatmDA_a00f9e8a Completed 3 Cycles: *SUCCESS* at Mon Oct 28 17:16:34 UTC 2024

@DavidHuber-NOAA
Copy link
Contributor

@jiaruidong2017 It looks like the dependencies are not quite right. The workflow stalled because the enkfgdas_esnowrecen job couldn't launch for cycle 2021122018. Running rocotocheck on this job shows

Task: enkfgdas_esnowrecen
...
  dependencies
    AND is not satisfied
      gdas_prepsnowcover of cycle 202112201800 is not SUCCEEDED
...

Task can not be submitted because:
  Dependencies are not satisfied

I assume the enkfgdas_esnowrecen job should only run on 00Z cycles, correct? If that's the case, then cycledef for the job probably needs to be changed to gdas_prep_snocvr from:

'cycledef': self.run.replace('enkf', ''),

to

'cycledef': 'gdas_prep_snocvr',

Also, the dependencies for the esfc job need to be reworked to add a task_valid test for esnowrecen similar to snowanl:

deps2 = []
dep_dict = {'type': 'taskvalid', 'name': f'{self.run}_prepsnowcover', 'condition': 'not'}
deps2.append(rocoto.add_dependency(dep_dict))
dep_dict = {'type': 'task', 'name': f'{self.run}_prep'}
deps2.append(rocoto.add_dependency(dep_dict))
deps.append(rocoto.create_dependency(dep_condition='and', dep=deps2))

@jiaruidong2017
Copy link
Contributor Author

@DavidHuber-NOAA The enkfgdas_esnowrecen job should run at all four cycles.

@DavidHuber-NOAA
Copy link
Contributor

OK, then the dependencies for just the esnowrecen job just need to be changed. They should include a taskvalid dependency similar to snowanl.

@jiaruidong2017
Copy link
Contributor Author

@DavidHuber-NOAA I have updated the esnowrecen job dependency. I just simply removed the prepsnowobs dependency, because this esnowrecen job doesn't require the prepsnowobs job dependency.

I didn't conduct the hybrid run in my test. Do you want me to run the hybrid test first or you submit a CI test directly? Please let me know. Thank you.

@jiaruidong2017
Copy link
Contributor Author

@DavidHuber-NOAA I have just submitted a hybrid test run, and will let you know when it is done.

@jiaruidong2017
Copy link
Contributor Author

@DavidHuber-NOAA The hybrid run succeeded as below. It is ready to have a CI test.

> rocotostat -d C96C48snowda_bufrtest4.db -w C96C48snowda_bufrtest4.xml

       CYCLE                    TASK                       JOBID               STATE         EXIT STATUS     TRIES      DURATION
================================================================================================================================
202112201200           gdas_stage_ic                     1677706           SUCCEEDED                   0         1          33.0
202112201200          gdas_fcst_seg0                     1678045           SUCCEEDED                   0         1         420.0
202112201200    gdas_atmos_prod_f000                     1678420           SUCCEEDED                   0         1          82.0
202112201200    gdas_atmos_prod_f001                     1678419           SUCCEEDED                   0         1          83.0
202112201200    gdas_atmos_prod_f002                     1678738           SUCCEEDED                   0         1          79.0
202112201200    gdas_atmos_prod_f003                     1678744           SUCCEEDED                   0         1          78.0
202112201200    gdas_atmos_prod_f004                     1678743           SUCCEEDED                   0         1          79.0
202112201200    gdas_atmos_prod_f005                     1678739           SUCCEEDED                   0         1          78.0
202112201200    gdas_atmos_prod_f006                     1678737           SUCCEEDED                   0         1          79.0
202112201200    gdas_atmos_prod_f007                     1678742           SUCCEEDED                   0         1          78.0
202112201200    gdas_atmos_prod_f008                     1678740           SUCCEEDED                   0         1          79.0
202112201200    gdas_atmos_prod_f009                     1678741           SUCCEEDED                   0         1          79.0
202112201200        enkfgdas_echgres                     1686305           SUCCEEDED                   0         1          32.0
202112201200       enkfgdas_stage_ic                     1685630           SUCCEEDED                   0         4          38.0
202112201200    enkfgdas_fcst_mem001                     1685784           SUCCEEDED                   0         1         337.0
202112201200    enkfgdas_fcst_mem002                     1685781           SUCCEEDED                   0         1         336.0
202112201200    enkfgdas_fcst_mem003                     1685776           SUCCEEDED                   0         1         321.0
202112201200    enkfgdas_fcst_mem004                     1685779           SUCCEEDED                   0         1         326.0
202112201200    enkfgdas_fcst_mem005                     1685780           SUCCEEDED                   0         1         325.0
202112201200    enkfgdas_fcst_mem006                     1685785           SUCCEEDED                   0         1         339.0
202112201200    enkfgdas_fcst_mem007                     1685778           SUCCEEDED                   0         1         315.0
202112201200    enkfgdas_fcst_mem008                     1685777           SUCCEEDED                   0         1         323.0
202112201200    enkfgdas_fcst_mem009                     1685782           SUCCEEDED                   0         1         333.0
202112201200    enkfgdas_fcst_mem010                     1685783           SUCCEEDED                   0         1         338.0
202112201200        enkfgdas_epos000                     1686304           SUCCEEDED                   0         1          34.0
202112201200        enkfgdas_epos001                     1686303           SUCCEEDED                   0         1          34.0
202112201200        enkfgdas_epos002                     1686306           SUCCEEDED                   0         1          34.0
202112201200        enkfgdas_epos003                     1686307           SUCCEEDED                   0         1          34.0
202112201200        enkfgdas_epos004                     1686308           SUCCEEDED                   0         1          33.0
202112201200        enkfgdas_epos005                     1686309           SUCCEEDED                   0         1          33.0
202112201200        enkfgdas_epos006                     1686310           SUCCEEDED                   0         1          34.0
================================================================================================================================
202112201800               gdas_prep                     1679185           SUCCEEDED                   0         1         256.0
202112201800               gdas_anal                     1686562           SUCCEEDED                   0         1         957.0
202112201800             gdas_sfcanl                     1687834           SUCCEEDED                   0         1          46.0
202112201800           gdas_analcalc                     1688262           SUCCEEDED                   0         1          78.0
202112201800            gdas_snowanl                     1680385           SUCCEEDED                   0         1         112.0
202112201800           gdas_analdiag                     1687800           SUCCEEDED                   0         1         125.0
202112201800          gdas_atmanlupp                     1688614           SUCCEEDED                   0         1          54.0
202112201800         gdas_atmanlprod                     1688752           SUCCEEDED                   0         1          75.0
202112201800          gdas_fcst_seg0                     1688263           SUCCEEDED                   0         1         509.0
202112201800    gdas_atmos_prod_f000                     1688618           SUCCEEDED                   0         1          79.0
202112201800    gdas_atmos_prod_f001                     1688613           SUCCEEDED                   0         1          82.0
202112201800    gdas_atmos_prod_f002                     1688759           SUCCEEDED                   0         1          82.0
202112201800    gdas_atmos_prod_f003                     1688758           SUCCEEDED                   0         1          77.0
202112201800    gdas_atmos_prod_f004                     1688753           SUCCEEDED                   0         1          79.0
202112201800    gdas_atmos_prod_f005                     1688754           SUCCEEDED                   0         1          79.0
202112201800    gdas_atmos_prod_f006                     1688793           SUCCEEDED                   0         1          77.0
202112201800    gdas_atmos_prod_f007                     1688757           SUCCEEDED                   0         1          80.0
202112201800    gdas_atmos_prod_f008                     1688770           SUCCEEDED                   0         1          78.0
202112201800    gdas_atmos_prod_f009                     1688767           SUCCEEDED                   0         1          83.0
202112201800            gdas_fit2obs                     1689020           SUCCEEDED                   0         1          20.0
202112201800            gdas_verfozn                     1688264           SUCCEEDED                   0         1          41.0
202112201800            gdas_verfrad                     1688265           SUCCEEDED                   0         1         533.0
202112201800            gdas_vminmon                     1687799           SUCCEEDED                   0         1          22.0
202112201800               gdas_arch                     1689204           SUCCEEDED                   0         1          89.0
202112201800            gdas_cleanup                     1689400           SUCCEEDED                   0         1          24.0
202112201800           enkfgdas_eobs                     1686567           SUCCEEDED                   0         1         464.0
202112201800           enkfgdas_eupd                     1687587           SUCCEEDED                   0         1         169.0
202112201800        enkfgdas_echgres                     1689179           SUCCEEDED                   0         1          35.0
202112201800          enkfgdas_ediag                     1687304           SUCCEEDED                   0         1         149.0
202112201800     enkfgdas_esnowrecen                     1686561           SUCCEEDED                   0         1         160.0
202112201800        enkfgdas_ecen000                     1688617           SUCCEEDED                   0         1          49.0
202112201800        enkfgdas_ecen001                     1688616           SUCCEEDED                   0         1          49.0
202112201800        enkfgdas_ecen002                     1688612           SUCCEEDED                   0         1          51.0
202112201800           enkfgdas_esfc                     1688626           SUCCEEDED                   0         1         193.0
202112201800    enkfgdas_fcst_mem001                     1688794           SUCCEEDED                   0         1         386.0
202112201800    enkfgdas_fcst_mem002                     1688762           SUCCEEDED                   0         1         390.0
202112201800    enkfgdas_fcst_mem003                     1688769           SUCCEEDED                   0         1         399.0
202112201800    enkfgdas_fcst_mem004                     1688790           SUCCEEDED                   0         1         391.0
202112201800    enkfgdas_fcst_mem005                     1688771           SUCCEEDED                   0         1         399.0
202112201800    enkfgdas_fcst_mem006                     1688774           SUCCEEDED                   0         1         395.0
202112201800    enkfgdas_fcst_mem007                     1688783           SUCCEEDED                   0         1         399.0
202112201800    enkfgdas_fcst_mem008                     1688778           SUCCEEDED                   0         1         391.0
202112201800    enkfgdas_fcst_mem009                     1688779           SUCCEEDED                   0         1         390.0
202112201800    enkfgdas_fcst_mem010                     1688780           SUCCEEDED                   0         1         388.0
202112201800        enkfgdas_epos000                     1689203           SUCCEEDED                   0         1          33.0
202112201800        enkfgdas_epos001                     1689181           SUCCEEDED                   0         1          33.0
202112201800        enkfgdas_epos002                     1689180           SUCCEEDED                   0         1          36.0
202112201800        enkfgdas_epos003                     1689205           SUCCEEDED                   0         1          34.0
202112201800        enkfgdas_epos004                     1689178           SUCCEEDED                   0         1          33.0
202112201800        enkfgdas_epos005                     1689207           SUCCEEDED                   0         1          35.0
202112201800        enkfgdas_epos006                     1689182           SUCCEEDED                   0         1          34.0
202112201800         enkfgdas_earc00                     1689391           SUCCEEDED                   0         1          48.0
202112201800         enkfgdas_earc01                     1689393           SUCCEEDED                   0         1          68.0
202112201800        enkfgdas_cleanup                     1689582           SUCCEEDED                   0         1          35.0
202112201800                gfs_prep                     1679190           SUCCEEDED                   0         1         249.0
202112201800                gfs_anal                     1686560           SUCCEEDED                   0         1         586.0
202112201800              gfs_sfcanl                     1687588           SUCCEEDED                   0         1          49.0
202112201800            gfs_analcalc                     1687801           SUCCEEDED                   0         1          78.0
202112201800             gfs_snowanl                     1680386           SUCCEEDED                   0         1         103.0
202112201800           gfs_atmanlupp                     1688266           SUCCEEDED                   0         1          51.0
202112201800          gfs_atmanlprod                     1688628           SUCCEEDED                   0         1         125.0
202112201800           gfs_fcst_seg0                     1687802           SUCCEEDED                   0         1        3917.0
202112201800     gfs_atmos_prod_f000                     1688378           SUCCEEDED                   0         1         132.0
202112201800     gfs_atmos_prod_f003                     1688377           SUCCEEDED                   0         1         135.0
202112201800     gfs_atmos_prod_f006                     1688379           SUCCEEDED                   0         1         130.0
202112201800     gfs_atmos_prod_f009                     1688621           SUCCEEDED                   0         1         131.0
202112201800     gfs_atmos_prod_f012                     1688620           SUCCEEDED                   0         1         128.0
202112201800     gfs_atmos_prod_f015                     1688619           SUCCEEDED                   0         1         129.0
202112201800     gfs_atmos_prod_f018                     1688622           SUCCEEDED                   0         1         129.0
202112201800     gfs_atmos_prod_f021                     1688787           SUCCEEDED                   0         1         129.0
202112201800     gfs_atmos_prod_f024                     1688789           SUCCEEDED                   0         1         129.0
202112201800     gfs_atmos_prod_f027                     1688786           SUCCEEDED                   0         1         130.0
202112201800     gfs_atmos_prod_f030                     1689010           SUCCEEDED                   0         1         129.0
202112201800     gfs_atmos_prod_f033                     1689015           SUCCEEDED                   0         1         129.0
202112201800     gfs_atmos_prod_f036                     1689009           SUCCEEDED                   0         1         129.0
202112201800     gfs_atmos_prod_f039                     1689183           SUCCEEDED                   0         1         129.0
202112201800     gfs_atmos_prod_f042                     1689184           SUCCEEDED                   0         1         127.0
202112201800     gfs_atmos_prod_f045                     1689185           SUCCEEDED                   0         1         129.0
202112201800     gfs_atmos_prod_f048                     1689392           SUCCEEDED                   0         1         130.0
202112201800     gfs_atmos_prod_f051                     1689399           SUCCEEDED                   0         1         129.0
202112201800     gfs_atmos_prod_f054                     1689394           SUCCEEDED                   0         1         131.0
202112201800     gfs_atmos_prod_f057                     1689395           SUCCEEDED                   0         1         130.0
202112201800     gfs_atmos_prod_f060                     1689583           SUCCEEDED                   0         1         142.0
202112201800     gfs_atmos_prod_f063                     1689584           SUCCEEDED                   0         1         128.0
202112201800     gfs_atmos_prod_f066                     1689586           SUCCEEDED                   0         1         130.0
202112201800     gfs_atmos_prod_f069                     1689817           SUCCEEDED                   0         1         140.0
202112201800     gfs_atmos_prod_f072                     1689818           SUCCEEDED                   0         1         141.0
202112201800     gfs_atmos_prod_f075                     1689819           SUCCEEDED                   0         1         137.0
202112201800     gfs_atmos_prod_f078                     1689981           SUCCEEDED                   0         1         134.0
202112201800     gfs_atmos_prod_f081                     1689951           SUCCEEDED                   0         1         131.0
202112201800     gfs_atmos_prod_f084                     1689950           SUCCEEDED                   0         1         134.0
202112201800     gfs_atmos_prod_f087                     1689957           SUCCEEDED                   0         1         132.0
202112201800     gfs_atmos_prod_f090                     1690169           SUCCEEDED                   0         1         133.0
202112201800     gfs_atmos_prod_f093                     1690168           SUCCEEDED                   0         1         132.0
202112201800     gfs_atmos_prod_f096                     1690172           SUCCEEDED                   0         1         129.0
202112201800     gfs_atmos_prod_f099                     1690275           SUCCEEDED                   0         1         129.0
202112201800     gfs_atmos_prod_f102                     1690274           SUCCEEDED                   0         1         128.0
202112201800     gfs_atmos_prod_f105                     1690276           SUCCEEDED                   0         1         127.0
202112201800     gfs_atmos_prod_f108                     1690545           SUCCEEDED                   0         1         129.0
202112201800     gfs_atmos_prod_f111                     1690554           SUCCEEDED                   0         1         129.0
202112201800     gfs_atmos_prod_f114                     1690548           SUCCEEDED                   0         1         129.0
202112201800     gfs_atmos_prod_f117                     1690739           SUCCEEDED                   0         1         130.0
202112201800     gfs_atmos_prod_f120                     1690740           SUCCEEDED                   0         1         133.0
202112201800             gfs_vminmon                     1687593           SUCCEEDED                   0         1          21.0
202112201800             gfs_tracker                     1690991           SUCCEEDED                   0         1          70.0
202112201800             gfs_genesis                     1690989           SUCCEEDED                   0         1         491.0
202112201800            gfs_metpg2g1                     1691467           SUCCEEDED                   0         1         121.0
202112201800            gfs_metpg2o1                     1691665           SUCCEEDED                   0         1         127.0
202112201800            gfs_metppcp1                     1691890           SUCCEEDED                   0         1          31.0
202112201800                gfs_arch                     1691357           SUCCEEDED                   0         1         242.0
202112201800             gfs_cleanup                     1692110           SUCCEEDED                   0         1          29.0
================================================================================================================================
202112210000               gdas_prep                     1689013           SUCCEEDED                   0         1         287.0
202112210000               gdas_anal                     1689396           SUCCEEDED                   0         1        1108.0
202112210000             gdas_sfcanl                     1690173           SUCCEEDED                   0         1          48.0
202112210000           gdas_analcalc                     1690277           SUCCEEDED                   0         1          82.0
202112210000      gdas_prepsnowcover                     1689397           SUCCEEDED                   0         1          71.0
202112210000            gdas_snowanl                     1689587           SUCCEEDED                   0         1         112.0
202112210000           gdas_analdiag                     1690174           SUCCEEDED                   0         1         132.0
202112210000          gdas_atmanlupp                     1690553           SUCCEEDED                   0         1          52.0
202112210000         gdas_atmanlprod                     1690741           SUCCEEDED                   0         1          77.0
202112210000          gdas_fcst_seg0                     1690278           SUCCEEDED                   0         1         503.0
202112210000    gdas_atmos_prod_f000                     1690742           SUCCEEDED                   0         1          80.0
202112210000    gdas_atmos_prod_f001                     1690743           SUCCEEDED                   0         1          79.0
202112210000    gdas_atmos_prod_f002                     1690744           SUCCEEDED                   0         1          82.0
202112210000    gdas_atmos_prod_f003                     1690745           SUCCEEDED                   0         1          85.0
202112210000    gdas_atmos_prod_f004                     1690746           SUCCEEDED                   0         1          84.0
202112210000    gdas_atmos_prod_f005                     1690747           SUCCEEDED                   0         1          82.0
202112210000    gdas_atmos_prod_f006                     1690748           SUCCEEDED                   0         1          81.0
202112210000    gdas_atmos_prod_f007                     1690749           SUCCEEDED                   0         1          84.0
202112210000    gdas_atmos_prod_f008                     1690750           SUCCEEDED                   0         1          80.0
202112210000    gdas_atmos_prod_f009                     1690751           SUCCEEDED                   0         1          81.0
202112210000            gdas_fit2obs                     1690994           SUCCEEDED                   0         1          20.0
202112210000            gdas_verfozn                     1690279           SUCCEEDED                   0         1          40.0
202112210000            gdas_verfrad                     1690280           SUCCEEDED                   0         1         648.0
202112210000            gdas_vminmon                     1690176           SUCCEEDED                   0         1          24.0
202112210000               gdas_arch                     1691148           SUCCEEDED                   0         1          70.0
202112210000            gdas_cleanup                     1691358           SUCCEEDED                   0         1          24.0
202112210000           enkfgdas_eobs                     1689398           SUCCEEDED                   0         1         476.0
202112210000           enkfgdas_eupd                     1689963           SUCCEEDED                   0         1         192.0
202112210000        enkfgdas_echgres                     1691146           SUCCEEDED                   0         1          32.0
202112210000          enkfgdas_ediag                     1689823           SUCCEEDED                   0         1         153.0
202112210000     enkfgdas_esnowrecen                     1689824           SUCCEEDED                   0         1         127.0
202112210000        enkfgdas_ecen000                     1690547           SUCCEEDED                   0         1          51.0
202112210000        enkfgdas_ecen001                     1690557           SUCCEEDED                   0         1          50.0
202112210000        enkfgdas_ecen002                     1690549           SUCCEEDED                   0         1          50.0
202112210000           enkfgdas_esfc                     1690551           SUCCEEDED                   0         1         187.0
202112210000    enkfgdas_fcst_mem001                     1690752           SUCCEEDED                   0         1         389.0
202112210000    enkfgdas_fcst_mem002                     1690753           SUCCEEDED                   0         1         386.0
202112210000    enkfgdas_fcst_mem003                     1690754           SUCCEEDED                   0         1         384.0
202112210000    enkfgdas_fcst_mem004                     1690755           SUCCEEDED                   0         1         384.0
202112210000    enkfgdas_fcst_mem005                     1690756           SUCCEEDED                   0         1         389.0
202112210000    enkfgdas_fcst_mem006                     1690757           SUCCEEDED                   0         1         385.0
202112210000    enkfgdas_fcst_mem007                     1690758           SUCCEEDED                   0         1         385.0
202112210000    enkfgdas_fcst_mem008                     1690759           SUCCEEDED                   0         1         382.0
202112210000    enkfgdas_fcst_mem009                     1690760           SUCCEEDED                   0         1         402.0
202112210000    enkfgdas_fcst_mem010                     1690761           SUCCEEDED                   0         1         393.0
202112210000        enkfgdas_epos000                     1691147           SUCCEEDED                   0         1          34.0
202112210000        enkfgdas_epos001                     1691149           SUCCEEDED                   0         1          34.0
202112210000        enkfgdas_epos002                     1691150           SUCCEEDED                   0         1          34.0
202112210000        enkfgdas_epos003                     1691151           SUCCEEDED                   0         1          34.0
202112210000        enkfgdas_epos004                     1691152           SUCCEEDED                   0         1          33.0
202112210000        enkfgdas_epos005                     1691153           SUCCEEDED                   0         1          33.0
202112210000        enkfgdas_epos006                     1691154           SUCCEEDED                   0         1          33.0
202112210000         enkfgdas_earc00                     1691359           SUCCEEDED                   0         1          41.0
202112210000         enkfgdas_earc01                     1691360           SUCCEEDED                   0         1          50.0
202112210000        enkfgdas_cleanup                     1691468           SUCCEEDED                   0         1          28.0
202112210000       gfs_prepsnowcover                           -                   -                   -         -             -
================================================================================================================================
202112210600               gdas_prep                     1690993           SUCCEEDED                   0         1         216.0
202112210600               gdas_anal                     1691465           SUCCEEDED                   0         1        1255.0
202112210600             gdas_sfcanl                     1692803           SUCCEEDED                   0         1          48.0
202112210600           gdas_analcalc                     1693001           SUCCEEDED                   0         1          81.0
202112210600            gdas_snowanl                     1691466           SUCCEEDED                   0         1          94.0
202112210600           gdas_analdiag                     1692804           SUCCEEDED                   0         1         128.0
202112210600          gdas_atmanlupp                     1693641           SUCCEEDED                   0         1          55.0
202112210600         gdas_atmanlprod                     1693916           SUCCEEDED                   0         1          77.0
202112210600          gdas_fcst_seg0                     1693003           SUCCEEDED                   0         1         514.0
202112210600    gdas_atmos_prod_f000                     1693640           SUCCEEDED                   0         1          78.0
202112210600    gdas_atmos_prod_f001                     1693642           SUCCEEDED                   0         1          80.0
202112210600    gdas_atmos_prod_f002                     1693643           SUCCEEDED                   0         1          80.0
202112210600    gdas_atmos_prod_f003                     1693690           SUCCEEDED                   0         1          85.0
202112210600    gdas_atmos_prod_f004                     1693689           SUCCEEDED                   0         1          81.0
202112210600    gdas_atmos_prod_f005                     1693644           SUCCEEDED                   0         1          79.0
202112210600    gdas_atmos_prod_f006                     1693645           SUCCEEDED                   0         1          78.0
202112210600    gdas_atmos_prod_f007                     1693646           SUCCEEDED                   0         1          78.0
202112210600    gdas_atmos_prod_f008                     1693691           SUCCEEDED                   0         1          80.0
202112210600    gdas_atmos_prod_f009                     1693647           SUCCEEDED                   0         1          80.0
202112210600            gdas_fit2obs                     1693918           SUCCEEDED                   0         1          21.0
202112210600            gdas_verfozn                     1693004           SUCCEEDED                   0         1          42.0
202112210600            gdas_verfrad                     1693005           SUCCEEDED                   0         1         647.0
202112210600            gdas_vminmon                     1692805           SUCCEEDED                   0         1          28.0
202112210600               gdas_arch                     1694017           SUCCEEDED                   0         1          63.0
202112210600            gdas_cleanup                     1694210           SUCCEEDED                   0         1          25.0
202112210600           enkfgdas_eobs                     1691469           SUCCEEDED                   0         1         501.0
202112210600           enkfgdas_eupd                     1692111           SUCCEEDED                   0         1         181.0
202112210600        enkfgdas_echgres                     1694211           SUCCEEDED                   0         1          32.0
202112210600          enkfgdas_ediag                     1691889           SUCCEEDED                   0         1         155.0
202112210600     enkfgdas_esnowrecen                     1691666           SUCCEEDED                   0         1         125.0
202112210600        enkfgdas_ecen000                     1693407           SUCCEEDED                   0         1          49.0
202112210600        enkfgdas_ecen001                     1693404           SUCCEEDED                   0         1          49.0
202112210600        enkfgdas_ecen002                     1693406           SUCCEEDED                   0         1          48.0
202112210600           enkfgdas_esfc                     1693405           SUCCEEDED                   0         1         190.0
202112210600    enkfgdas_fcst_mem001                     1693915           SUCCEEDED                   0         1         401.0
202112210600    enkfgdas_fcst_mem002                     1693920           SUCCEEDED                   0         1         395.0
202112210600    enkfgdas_fcst_mem003                     1693921           SUCCEEDED                   0         1         395.0
202112210600    enkfgdas_fcst_mem004                     1693922           SUCCEEDED                   0         1         390.0
202112210600    enkfgdas_fcst_mem005                     1693924           SUCCEEDED                   0         1         388.0
202112210600    enkfgdas_fcst_mem006                     1693923           SUCCEEDED                   0         1         386.0
202112210600    enkfgdas_fcst_mem007                     1693928           SUCCEEDED                   0         1         384.0
202112210600    enkfgdas_fcst_mem008                     1693926           SUCCEEDED                   0         1         387.0
202112210600    enkfgdas_fcst_mem009                     1693929           SUCCEEDED                   0         1         386.0
202112210600    enkfgdas_fcst_mem010                     1693932           SUCCEEDED                   0         1         384.0
202112210600        enkfgdas_epos000                     1694212           SUCCEEDED                   0         1          33.0
202112210600        enkfgdas_epos001                     1694213           SUCCEEDED                   0         1          34.0
202112210600        enkfgdas_epos002                     1694214           SUCCEEDED                   0         1          34.0
202112210600        enkfgdas_epos003                     1694215           SUCCEEDED                   0         1          33.0
202112210600        enkfgdas_epos004                     1694216           SUCCEEDED                   0         1          33.0
202112210600        enkfgdas_epos005                     1694217           SUCCEEDED                   0         1          33.0
202112210600        enkfgdas_epos006                     1694218           SUCCEEDED                   0         1          34.0
202112210600         enkfgdas_earc00                     1694330           SUCCEEDED                   0         1          43.0
202112210600         enkfgdas_earc01                     1694331           SUCCEEDED                   0         1          37.0
202112210600        enkfgdas_cleanup                     1694539           SUCCEEDED                   0         1          26.0

@DavidHuber-NOAA
Copy link
Contributor

@jiaruidong2017 I see that there is a gfs_prepsnowcover task for the 2021122100 cycle. This job will never run and will eventually cause longer running experiments to stall as the cycle will remain 'active'. There shouldn't be a gfs prepsnowcover job, correct?

@jiaruidong2017
Copy link
Contributor Author

@DavidHuber-NOAA The prepsnowcover job is designed only for the IMS data preprocessing at 00Z, and the IMS data is designed to assimilate only at the 00Z cycle.

Previously, the gfs run is at 00z, while the above gfs run in CI test is at 18z. Therefore, the current design of IMS data processing and DA at 00z will be never happened for the gfs runs. @CoryMartin-NOAA Do we need to make change to move IMS data processing and DA from 00Z to 18Z?

@CoryMartin-NOAA
Copy link
Contributor

@jiaruidong2017 no, for this CI test, if the GFS cycle is only at 18z, then there is no gfs_prepsnowcover. This is okay, since there will be a gdas_prepsnowcover that is tested. The 18z cycles for both GFS and GDAS will just assimilate in situ snow observations.

@jiaruidong2017
Copy link
Contributor Author

Thanks @CoryMartin-NOAA for clarifying this.

Once the gfs cycle is running at 00z cycle, the gfs_prepsnowcover job will be used to process IMS data. The gfs_snowanl is the only job depending on the gfs_prepsnowcover job. If the gfs cycle is not running at 00z cycle, both gfs_prepsnowcover and gfs_snowanl jobs will not start. Therefore, although this job will never run, this job will never stall longer running experiments. Therefore, I think this PR is ready to proceed with CI test. @DavidHuber-NOAA @CoryMartin-NOAA do you agree?

@CoryMartin-NOAA
Copy link
Contributor

if there's no GFS forecast, there should be no GFS prepsnowcover job, but it seems like there is one showing up, right? That needs to be fixed

@jiaruidong2017
Copy link
Contributor Author

@DavidHuber-NOAA I updated the gfs_prepsnowcover job by using cycledef=gfs. I have conducted the test run, and the hybrid run succeeded as below. Would you please review it. Thanks.

rocotostat -d C96C48snowda_bufrtest4.db -w C96C48snowda_bufrtest4.xml

       CYCLE                    TASK                       JOBID               STATE         EXIT STATUS     TRIES      DURATION
================================================================================================================================
202112201200           gdas_stage_ic                     1746493           SUCCEEDED                   0         1          32.0
202112201200          gdas_fcst_seg0                     1746690           SUCCEEDED                   0         1         446.0
202112201200    gdas_atmos_prod_f000                     1746969           SUCCEEDED                   0         1          81.0
202112201200    gdas_atmos_prod_f001                     1746971           SUCCEEDED                   0         1          80.0
202112201200    gdas_atmos_prod_f002                     1747131           SUCCEEDED                   0         1          87.0
202112201200    gdas_atmos_prod_f003                     1747118           SUCCEEDED                   0         1          83.0
202112201200    gdas_atmos_prod_f004                     1747119           SUCCEEDED                   0         1          85.0
202112201200    gdas_atmos_prod_f005                     1747120           SUCCEEDED                   0         1          82.0
202112201200    gdas_atmos_prod_f006                     1747121           SUCCEEDED                   0         1          82.0
202112201200    gdas_atmos_prod_f007                     1747122           SUCCEEDED                   0         1          84.0
202112201200    gdas_atmos_prod_f008                     1747123           SUCCEEDED                   0         1          85.0
202112201200    gdas_atmos_prod_f009                     1747126           SUCCEEDED                   0         1          88.0
202112201200        enkfgdas_echgres                     1747128           SUCCEEDED                   0         1          38.0
202112201200       enkfgdas_stage_ic                     1746494           SUCCEEDED                   0         1          40.0
202112201200    enkfgdas_fcst_mem001                     1746691           SUCCEEDED                   0         1         342.0
202112201200    enkfgdas_fcst_mem002                     1746692           SUCCEEDED                   0         1         338.0
202112201200    enkfgdas_fcst_mem003                     1746693           SUCCEEDED                   0         1         335.0
202112201200    enkfgdas_fcst_mem004                     1746694           SUCCEEDED                   0         1         341.0
202112201200    enkfgdas_fcst_mem005                     1746695           SUCCEEDED                   0         1         339.0
202112201200    enkfgdas_fcst_mem006                     1746696           SUCCEEDED                   0         1         350.0
202112201200    enkfgdas_fcst_mem007                     1746697           SUCCEEDED                   0         1         331.0
202112201200    enkfgdas_fcst_mem008                     1746698           SUCCEEDED                   0         1         335.0
202112201200    enkfgdas_fcst_mem009                     1746699           SUCCEEDED                   0         1         333.0
202112201200    enkfgdas_fcst_mem010                     1746704           SUCCEEDED                   0         1         334.0
202112201200        enkfgdas_epos000                     1747129           SUCCEEDED                   0         1          36.0
202112201200        enkfgdas_epos001                     1747153           SUCCEEDED                   0         1          46.0
202112201200        enkfgdas_epos002                     1747133           SUCCEEDED                   0         1          38.0
202112201200        enkfgdas_epos003                     1747140           SUCCEEDED                   0         1          36.0
202112201200        enkfgdas_epos004                     1747137           SUCCEEDED                   0         1          37.0
202112201200        enkfgdas_epos005                     1747145           SUCCEEDED                   0         1          42.0
202112201200        enkfgdas_epos006                     1747147           SUCCEEDED                   0         1          45.0
================================================================================================================================
202112201800               gdas_prep                     1747509           SUCCEEDED                   0         1         265.0
202112201800               gdas_anal                     1747984           SUCCEEDED                   0         1         978.0
202112201800             gdas_sfcanl                     1748951           SUCCEEDED                   0         1          57.0
202112201800           gdas_analcalc                     1749153           SUCCEEDED                   0         1         168.0
202112201800            gdas_snowanl                     1747985           SUCCEEDED                   0         1         157.0
202112201800           gdas_analdiag                     1748948           SUCCEEDED                   0         1         134.0
202112201800          gdas_atmanlupp                     1749470           SUCCEEDED                   0         1          77.0
202112201800         gdas_atmanlprod                     1749837           SUCCEEDED                   0         1          78.0
202112201800          gdas_fcst_seg0                     1749154           SUCCEEDED                   0         1         575.0
202112201800    gdas_atmos_prod_f000                     1749836           SUCCEEDED                   0         1          82.0
202112201800    gdas_atmos_prod_f001                     1749796           SUCCEEDED                   0         1          81.0
202112201800    gdas_atmos_prod_f002                     1749797           SUCCEEDED                   0         1          82.0
202112201800    gdas_atmos_prod_f003                     1749798           SUCCEEDED                   0         1          82.0
202112201800    gdas_atmos_prod_f004                     1749799           SUCCEEDED                   0         1          82.0
202112201800    gdas_atmos_prod_f005                     1749800           SUCCEEDED                   0         1          81.0
202112201800    gdas_atmos_prod_f006                     1749801           SUCCEEDED                   0         1          82.0
202112201800    gdas_atmos_prod_f007                     1749802           SUCCEEDED                   0         1          83.0
202112201800    gdas_atmos_prod_f008                     1749803           SUCCEEDED                   0         1          81.0
202112201800    gdas_atmos_prod_f009                     1749804           SUCCEEDED                   0         1          82.0
202112201800            gdas_fit2obs                     1750267           SUCCEEDED                   0         1          23.0
202112201800            gdas_verfozn                     1749155           SUCCEEDED                   0         1          74.0
202112201800            gdas_verfrad                     1749156           SUCCEEDED                   0         1         603.0
202112201800            gdas_vminmon                     1748949           SUCCEEDED                   0         1          26.0
202112201800               gdas_arch                     1750506           SUCCEEDED                   0         1         100.0
202112201800            gdas_cleanup                     1750743           SUCCEEDED                   0         1          25.0
202112201800           enkfgdas_eobs                     1747989           SUCCEEDED                   0         1         488.0
202112201800           enkfgdas_eupd                     1748593           SUCCEEDED                   0         1         204.0
202112201800        enkfgdas_echgres                     1750507           SUCCEEDED                   0         1          38.0
202112201800          enkfgdas_ediag                     1748383           SUCCEEDED                   0         1         152.0
202112201800     enkfgdas_esnowrecen                     1748122           SUCCEEDED                   0         1         165.0
202112201800        enkfgdas_ecen000                     1749472           SUCCEEDED                   0         1          50.0
202112201800        enkfgdas_ecen001                     1749473           SUCCEEDED                   0         1          51.0
202112201800        enkfgdas_ecen002                     1749474           SUCCEEDED                   0         1          50.0
202112201800           enkfgdas_esfc                     1749475           SUCCEEDED                   0         1         222.0
202112201800    enkfgdas_fcst_mem001                     1749806           SUCCEEDED                   0         1         421.0
202112201800    enkfgdas_fcst_mem002                     1749807           SUCCEEDED                   0         1         425.0
202112201800    enkfgdas_fcst_mem003                     1749809           SUCCEEDED                   0         1         417.0
202112201800    enkfgdas_fcst_mem004                     1749813           SUCCEEDED                   0         1         416.0
202112201800    enkfgdas_fcst_mem005                     1749812           SUCCEEDED                   0         1         420.0
202112201800    enkfgdas_fcst_mem006                     1749871           SUCCEEDED                   0         1         425.0
202112201800    enkfgdas_fcst_mem007                     1749816           SUCCEEDED                   0         1         418.0
202112201800    enkfgdas_fcst_mem008                     1749824           SUCCEEDED                   0         1         425.0
202112201800    enkfgdas_fcst_mem009                     1749818           SUCCEEDED                   0         1         415.0
202112201800    enkfgdas_fcst_mem010                     1749823           SUCCEEDED                   0         1         415.0
202112201800        enkfgdas_epos000                     1750508           SUCCEEDED                   0         1          36.0
202112201800        enkfgdas_epos001                     1750509           SUCCEEDED                   0         1          37.0
202112201800        enkfgdas_epos002                     1750510           SUCCEEDED                   0         1          37.0
202112201800        enkfgdas_epos003                     1750511           SUCCEEDED                   0         1          37.0
202112201800        enkfgdas_epos004                     1750512           SUCCEEDED                   0         1          36.0
202112201800        enkfgdas_epos005                     1750513           SUCCEEDED                   0         1          35.0
202112201800        enkfgdas_epos006                     1750514           SUCCEEDED                   0         1          35.0
202112201800         enkfgdas_earc00                     1750744           SUCCEEDED                   0         1          53.0
202112201800         enkfgdas_earc01                     1750746           SUCCEEDED                   0         1          73.0
202112201800        enkfgdas_cleanup                     1750898           SUCCEEDED                   0         1          26.0
202112201800                gfs_prep                     1747508           SUCCEEDED                   0         1         268.0
202112201800                gfs_anal                     1747987           SUCCEEDED                   0         1         610.0
202112201800              gfs_sfcanl                     1748968           SUCCEEDED                   0         1          54.0
202112201800            gfs_analcalc                     1749157           SUCCEEDED                   0         1         168.0
202112201800       gfs_prepsnowcover                     1748447           SUCCEEDED                   0         4          67.0
202112201800             gfs_snowanl                     1748594           SUCCEEDED                   0         1         104.0
202112201800           gfs_atmanlupp                     1749479           SUCCEEDED                   0         1          77.0
202112201800          gfs_atmanlprod                     1749825           SUCCEEDED                   0         1         127.0
202112201800           gfs_fcst_seg0                     1749158           SUCCEEDED                   0         1        3998.0
202112201800     gfs_atmos_prod_f000                     1749827           SUCCEEDED                   0         1         131.0
202112201800     gfs_atmos_prod_f003                     1750268           SUCCEEDED                   0         1         151.0
202112201800     gfs_atmos_prod_f006                     1750269           SUCCEEDED                   0         1         152.0
202112201800     gfs_atmos_prod_f009                     1750270           SUCCEEDED                   0         1         142.0
202112201800     gfs_atmos_prod_f012                     1750271           SUCCEEDED                   0         1         149.0
202112201800     gfs_atmos_prod_f015                     1750272           SUCCEEDED                   0         1         147.0
202112201800     gfs_atmos_prod_f018                     1750515           SUCCEEDED                   0         1         139.0
202112201800     gfs_atmos_prod_f021                     1750516           SUCCEEDED                   0         1         135.0
202112201800     gfs_atmos_prod_f024                     1750517           SUCCEEDED                   0         1         142.0
202112201800     gfs_atmos_prod_f027                     1750749           SUCCEEDED                   0         1         134.0
202112201800     gfs_atmos_prod_f030                     1750755           SUCCEEDED                   0         1         133.0
202112201800     gfs_atmos_prod_f033                     1750756           SUCCEEDED                   0         1         139.0
202112201800     gfs_atmos_prod_f036                     1750761           SUCCEEDED                   0         1         136.0
202112201800     gfs_atmos_prod_f039                     1750897           SUCCEEDED                   0         1         136.0
202112201800     gfs_atmos_prod_f042                     1750902           SUCCEEDED                   0         1         143.0
202112201800     gfs_atmos_prod_f045                     1750904           SUCCEEDED                   0         1         136.0
202112201800     gfs_atmos_prod_f048                     1751414           SUCCEEDED                   0         1         143.0
202112201800     gfs_atmos_prod_f051                     1751413           SUCCEEDED                   0         1         137.0
202112201800     gfs_atmos_prod_f054                     1751415           SUCCEEDED                   0         1         136.0
202112201800     gfs_atmos_prod_f057                     1751774           SUCCEEDED                   0         1         132.0
202112201800     gfs_atmos_prod_f060                     1751775           SUCCEEDED                   0         1         139.0
202112201800     gfs_atmos_prod_f063                     1751776           SUCCEEDED                   0         1         136.0
202112201800     gfs_atmos_prod_f066                     1752023           SUCCEEDED                   0         1         135.0
202112201800     gfs_atmos_prod_f069                     1752024           SUCCEEDED                   0         1         134.0
202112201800     gfs_atmos_prod_f072                     1752025           SUCCEEDED                   0         1         135.0
202112201800     gfs_atmos_prod_f075                     1752027           SUCCEEDED                   0         1         135.0
202112201800     gfs_atmos_prod_f078                     1752159           SUCCEEDED                   0         1         133.0
202112201800     gfs_atmos_prod_f081                     1752161           SUCCEEDED                   0         1         134.0
202112201800     gfs_atmos_prod_f084                     1752162           SUCCEEDED                   0         1         137.0
202112201800     gfs_atmos_prod_f087                     1752374           SUCCEEDED                   0         1         134.0
202112201800     gfs_atmos_prod_f090                     1752373           SUCCEEDED                   0         1         137.0
202112201800     gfs_atmos_prod_f093                     1752375           SUCCEEDED                   0         1         140.0
202112201800     gfs_atmos_prod_f096                     1752498           SUCCEEDED                   0         1         135.0
202112201800     gfs_atmos_prod_f099                     1752499           SUCCEEDED                   0         1         140.0
202112201800     gfs_atmos_prod_f102                     1752501           SUCCEEDED                   0         1         134.0
202112201800     gfs_atmos_prod_f105                     1752746           SUCCEEDED                   0         1         133.0
202112201800     gfs_atmos_prod_f108                     1752773           SUCCEEDED                   0         1         131.0
202112201800     gfs_atmos_prod_f111                     1752761           SUCCEEDED                   0         1         132.0
202112201800     gfs_atmos_prod_f114                     1752859           SUCCEEDED                   0         1         134.0
202112201800     gfs_atmos_prod_f117                     1752853           SUCCEEDED                   0         1         137.0
202112201800     gfs_atmos_prod_f120                     1752852           SUCCEEDED                   0         1         131.0
202112201800             gfs_vminmon                     1748596           SUCCEEDED                   0         1          22.0
202112201800             gfs_tracker                     1753030           SUCCEEDED                   0         1          87.0
202112201800             gfs_genesis                     1753028           SUCCEEDED                   0         1         533.0
202112201800            gfs_metpg2g1                     1753627           SUCCEEDED                   0         1         164.0
202112201800            gfs_metpg2o1                     1753693           SUCCEEDED                   0         1         180.0
202112201800            gfs_metppcp1                     1753935           SUCCEEDED                   0         1          31.0
202112201800                gfs_arch                     1753415           SUCCEEDED                   0         1         368.0
202112201800             gfs_cleanup                     1754039           SUCCEEDED                   0         1          31.0
================================================================================================================================
202112210000               gdas_prep                     1750283           SUCCEEDED                   0         1         317.0
202112210000               gdas_anal                     1750762           SUCCEEDED                   0         1        1168.0
202112210000             gdas_sfcanl                     1752028           SUCCEEDED                   0         1          49.0
202112210000           gdas_analcalc                     1752166           SUCCEEDED                   0         1         125.0
202112210000      gdas_prepsnowcover                     1750763           SUCCEEDED                   0         1          69.0
202112210000            gdas_snowanl                     1750906           SUCCEEDED                   0         1         161.0
202112210000           gdas_analdiag                     1752029           SUCCEEDED                   0         1         136.0
202112210000          gdas_atmanlupp                     1752376           SUCCEEDED                   0         1          85.0
202112210000         gdas_atmanlprod                     1752504           SUCCEEDED                   0         1          79.0
202112210000          gdas_fcst_seg0                     1752168           SUCCEEDED                   0         1         560.0
202112210000    gdas_atmos_prod_f000                     1752505           SUCCEEDED                   0         1          83.0
202112210000    gdas_atmos_prod_f001                     1752506           SUCCEEDED                   0         1          79.0
202112210000    gdas_atmos_prod_f002                     1752503           SUCCEEDED                   0         1          83.0
202112210000    gdas_atmos_prod_f003                     1752510           SUCCEEDED                   0         1          83.0
202112210000    gdas_atmos_prod_f004                     1752507           SUCCEEDED                   0         1          83.0
202112210000    gdas_atmos_prod_f005                     1752508           SUCCEEDED                   0         1          81.0
202112210000    gdas_atmos_prod_f006                     1752511           SUCCEEDED                   0         1          84.0
202112210000    gdas_atmos_prod_f007                     1752512           SUCCEEDED                   0         1          82.0
202112210000    gdas_atmos_prod_f008                     1752513           SUCCEEDED                   0         1          82.0
202112210000    gdas_atmos_prod_f009                     1752514           SUCCEEDED                   0         1          82.0
202112210000            gdas_fit2obs                     1752767           SUCCEEDED                   0         1          21.0
202112210000            gdas_verfozn                     1752170           SUCCEEDED                   0         1          42.0
202112210000            gdas_verfrad                     1752171           SUCCEEDED                   0         1         703.0
202112210000            gdas_vminmon                     1752030           SUCCEEDED                   0         1          29.0
202112210000               gdas_arch                     1752857           SUCCEEDED                   0         1          81.0
202112210000            gdas_cleanup                     1753029           SUCCEEDED                   0         1          25.0
202112210000           enkfgdas_eobs                     1750764           SUCCEEDED                   0         1         482.0
202112210000           enkfgdas_eupd                     1751781           SUCCEEDED                   0         1         191.0
202112210000        enkfgdas_echgres                     1752854           SUCCEEDED                   0         1          37.0
202112210000          enkfgdas_ediag                     1751417           SUCCEEDED                   0         1         157.0
202112210000     enkfgdas_esnowrecen                     1751416           SUCCEEDED                   0         1         199.0
202112210000        enkfgdas_ecen000                     1752378           SUCCEEDED                   0         1          57.0
202112210000        enkfgdas_ecen001                     1752379           SUCCEEDED                   0         1          55.0
202112210000        enkfgdas_ecen002                     1752384           SUCCEEDED                   0         1          55.0
202112210000           enkfgdas_esfc                     1752385           SUCCEEDED                   0         1         213.0
202112210000    enkfgdas_fcst_mem001                     1752515           SUCCEEDED                   0         1         412.0
202112210000    enkfgdas_fcst_mem002                     1752516           SUCCEEDED                   0         1         421.0
202112210000    enkfgdas_fcst_mem003                     1752517           SUCCEEDED                   0         1         408.0
202112210000    enkfgdas_fcst_mem004                     1752518           SUCCEEDED                   0         1         417.0
202112210000    enkfgdas_fcst_mem005                     1752519           SUCCEEDED                   0         1         415.0
202112210000    enkfgdas_fcst_mem006                     1752520           SUCCEEDED                   0         1         410.0
202112210000    enkfgdas_fcst_mem007                     1752522           SUCCEEDED                   0         1         414.0
202112210000    enkfgdas_fcst_mem008                     1752521           SUCCEEDED                   0         1         409.0
202112210000    enkfgdas_fcst_mem009                     1752523           SUCCEEDED                   0         1         405.0
202112210000    enkfgdas_fcst_mem010                     1752768           SUCCEEDED                   0         1         392.0
202112210000        enkfgdas_epos000                     1753031           SUCCEEDED                   0         1          35.0
202112210000        enkfgdas_epos001                     1753032           SUCCEEDED                   0         1          34.0
202112210000        enkfgdas_epos002                     1753033           SUCCEEDED                   0         1          35.0
202112210000        enkfgdas_epos003                     1753034           SUCCEEDED                   0         1          36.0
202112210000        enkfgdas_epos004                     1753035           SUCCEEDED                   0         1          34.0
202112210000        enkfgdas_epos005                     1753036           SUCCEEDED                   0         1          34.0
202112210000        enkfgdas_epos006                     1753038           SUCCEEDED                   0         1          34.0
202112210000         enkfgdas_earc00                     1753181           SUCCEEDED                   0         1          47.0
202112210000         enkfgdas_earc01                     1753182           SUCCEEDED                   0         1          63.0
202112210000        enkfgdas_cleanup                     1753416           SUCCEEDED                   0         1          26.0
================================================================================================================================
202112210600               gdas_prep                     1752766           SUCCEEDED                   0         1         245.0
202112210600               gdas_anal                     1753183           SUCCEEDED                   0         1        1276.0
202112210600             gdas_sfcanl                     1754237           SUCCEEDED                   0         1          48.0
202112210600           gdas_analcalc                     1754336           SUCCEEDED                   0         1          96.0
202112210600            gdas_snowanl                     1753037           SUCCEEDED                   0         1         132.0
202112210600           gdas_analdiag                     1754238           SUCCEEDED                   0         1         132.0
202112210600          gdas_atmanlupp                     1754575           SUCCEEDED                   0         1          57.0
202112210600         gdas_atmanlprod                     1754744           SUCCEEDED                   0         1          77.0
202112210600          gdas_fcst_seg0                     1754337           SUCCEEDED                   0         1         532.0
202112210600    gdas_atmos_prod_f000                     1754576           SUCCEEDED                   0         1          79.0
202112210600    gdas_atmos_prod_f001                     1754577           SUCCEEDED                   0         1          82.0
202112210600    gdas_atmos_prod_f002                     1754578           SUCCEEDED                   0         1          79.0
202112210600    gdas_atmos_prod_f003                     1754579           SUCCEEDED                   0         1          79.0
202112210600    gdas_atmos_prod_f004                     1754580           SUCCEEDED                   0         1          79.0
202112210600    gdas_atmos_prod_f005                     1754745           SUCCEEDED                   0         1          78.0
202112210600    gdas_atmos_prod_f006                     1754742           SUCCEEDED                   0         1          80.0
202112210600    gdas_atmos_prod_f007                     1754746           SUCCEEDED                   0         1          78.0
202112210600    gdas_atmos_prod_f008                     1754748           SUCCEEDED                   0         1          80.0
202112210600    gdas_atmos_prod_f009                     1754747           SUCCEEDED                   0         1          80.0
202112210600            gdas_fit2obs                     1754908           SUCCEEDED                   0         1          22.0
202112210600            gdas_verfozn                     1754338           SUCCEEDED                   0         1          42.0
202112210600            gdas_verfrad                     1754339           SUCCEEDED                   0         1         667.0
202112210600            gdas_vminmon                     1754239           SUCCEEDED                   0         1          22.0
202112210600               gdas_arch                     1755109           SUCCEEDED                   0         1          68.0
202112210600            gdas_cleanup                     1755195           SUCCEEDED                   0         1          25.0
202112210600           enkfgdas_eobs                     1753184           SUCCEEDED                   0         1         528.0
202112210600           enkfgdas_eupd                     1753694           SUCCEEDED                   0         1         201.0
202112210600        enkfgdas_echgres                     1754905           SUCCEEDED                   0         1          46.0
202112210600          enkfgdas_ediag                     1753628           SUCCEEDED                   0         1         156.0
202112210600     enkfgdas_esnowrecen                     1753185           SUCCEEDED                   0         1         156.0
202112210600        enkfgdas_ecen000                     1754493           SUCCEEDED                   0         1          48.0
202112210600        enkfgdas_ecen001                     1754494           SUCCEEDED                   0         1          53.0
202112210600        enkfgdas_ecen002                     1754495           SUCCEEDED                   0         1          48.0
202112210600           enkfgdas_esfc                     1754496           SUCCEEDED                   0         1         196.0
202112210600    enkfgdas_fcst_mem001                     1754584           SUCCEEDED                   0         1         409.0
202112210600    enkfgdas_fcst_mem002                     1754586           SUCCEEDED                   0         1         409.0
202112210600    enkfgdas_fcst_mem003                     1754589           SUCCEEDED                   0         1         395.0
202112210600    enkfgdas_fcst_mem004                     1754590           SUCCEEDED                   0         1         407.0
202112210600    enkfgdas_fcst_mem005                     1754591           SUCCEEDED                   0         1         406.0
202112210600    enkfgdas_fcst_mem006                     1754592           SUCCEEDED                   0         1         404.0
202112210600    enkfgdas_fcst_mem007                     1754593           SUCCEEDED                   0         1         399.0
202112210600    enkfgdas_fcst_mem008                     1754594           SUCCEEDED                   0         1         400.0
202112210600    enkfgdas_fcst_mem009                     1754595           SUCCEEDED                   0         1         397.0
202112210600    enkfgdas_fcst_mem010                     1754596           SUCCEEDED                   0         1         406.0
202112210600        enkfgdas_epos000                     1754904           SUCCEEDED                   0         1          35.0
202112210600        enkfgdas_epos001                     1754915           SUCCEEDED                   0         1          39.0
202112210600        enkfgdas_epos002                     1754903           SUCCEEDED                   0         1          38.0
202112210600        enkfgdas_epos003                     1754902           SUCCEEDED                   0         1          39.0
202112210600        enkfgdas_epos004                     1754910           SUCCEEDED                   0         1          34.0
202112210600        enkfgdas_epos005                     1754912           SUCCEEDED                   0         1          37.0
202112210600        enkfgdas_epos006                     1754911           SUCCEEDED                   0         1          37.0
202112210600         enkfgdas_earc00                     1755107           SUCCEEDED                   0         1          53.0
202112210600         enkfgdas_earc01                     1755111           SUCCEEDED                   0         1          54.0
202112210600        enkfgdas_cleanup                     1755194           SUCCEEDED                   0         1          27.0

Comment on lines 573 to 583
cycledef = 'gdas_prep_snocvr'
if self.run in ['gfs']:
cycledef = self.run

resources = self.get_resource('prepsnowcover')
task_name = f'{self.run}_prepsnowcover'
task_dict = {'task_name': task_name,
'resources': resources,
'dependency': dependencies,
'envars': self.envars,
'cycledef': 'gdas_prep_snocvr',
'cycledef': cycledef,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this solution will only work if the gfs runs on just the 00z cycle. If the gfs forecast is to run on 06, 12, and/or 18z, then this job will be added to the mesh and will fail. I see two options.

  1. Add another cycledef: gfs_prep_snocvr. The cycledef could then be defined here as f"{run}_prep_snocvr"
  2. Add an if block:
    (pseudocode)
if run == 'gfs':
   <add a taskvalid dependency for gdas_prepsnowcover>
   <cycledef = 'gfs'>
else:
   <cycledef = 'gdas_prep_snocvr'>

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @DavidHuber-NOAA for your review. Per above test, the gfs runs at 18z cycle, the added gfs_prepsnowcover job did nothing and will not stall the workflow. If the gfs forecast is to run on 06, 12, and/or 18z, the script scripts/exglobal_prep_snowcover.py will control gfs_prepsnowcover job doing nothing as below:

    if SnowAnl.task_config.cyc == 0:
        SnowAnl.prepare_IMS()

Therefore, the current settings won't cause any problems to the workflow.

I was thinking to add another cycledef: gfs_prep_snocvr, but I don't think it helps, because the gfs_prep_snocvr should be the same as gdas_prep_snocvr.

In addition, the option 2 you proposed above is the same with current setting as below:

        cycledef = 'gdas_prep_snocvr'
        if self.run in ['gfs']:
            cycledef = self.run

@DavidHuber-NOAA Do you agree with me? Thank you very much.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I do agree with you now that I see the change you made in exglobal_prep_snowcover.py. I think we would prefer to not have do-nothing jobs, but this would work for now.

@WalterKolczynski-NOAA what do you think?

Copy link
Contributor

@WalterKolczynski-NOAA WalterKolczynski-NOAA Oct 31, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree that we want to eliminate do-nothing jobs.

For a solution, we need to answer some questions first::

  • Is the goal to have the GFS job run once per day? Or should it really only run at 00z?
  • Can it run every day at 00z even if there is no GFS otherwise in that cycle (assuming there are cycles with GFS)?
  • What about if there is no GFS cycle on a given day? (For instance, GFS is running every 48 h)
  • What is actually being done by this job in the GFS cycle that is different than GDAS? Does it need to be run again for GFS at all?

We had similar restraints with metp, and the answers were once a day and it could be run even if there was no GFS in that cycle, but only if there was GFS sometime that day. The solution there was to run metp at 18z every day if the GFS ran more frequently than once a day (even if GFS did not run at 18z), and on the gfs cycledef if the GFS was run less often than once/day.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @WalterKolczynski-NOAA for your reply. I think @CoryMartin-NOAA is best person to give more accurate answers.

@CoryMartin-NOAA Would you please answer these questions? Thanks.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd need to check to confirm, but I think, while it doesn't need to run for GFS and would be identical to GDAS, the GFS cycle runs 'first', so in ops this matters, but in R&D not so much. So perhaps we can use the GDAS 00z file in the GFS 00z analysis (but I don't know if this dependency is a good idea).

For your other questions:

  • only at 00z, it's based on a L3 product
  • yes, waste of resources but no harm otherwise
  • either can skip it completely or run and produce a useless file
  • See above for answer, it's identical, but dependencies could be tricky

@DavidHuber-NOAA DavidHuber-NOAA removed the CI-Wcoss2-Running **Bot use only** CI testing on WCOSS for this PR is in-progress label Oct 30, 2024
@CoryMartin-NOAA
Copy link
Contributor

@WalterKolczynski-NOAA with all of this discussion, do you think it makes sense to instead wrap this job into the snowanl job and it will only process the snow cover obs if the cycle is 00z? That would be simpler from a workflow perspective, but might be a bit of a waste of resources. Any thoughts on that? @jiaruidong2017 how long does the IMS preprocessing take and is it only on 6 processors?

@jiaruidong2017
Copy link
Contributor Author

@CoryMartin-NOAA The prepsnowcover job is designed to use a single processor and take less than 5 minutes.

@WalterKolczynski-NOAA
Copy link
Contributor

Discussed this with Cory. We've decided to go forward with his proposed plan to just add the snow obs processing to the snow analysis job, since it is fast and doesn't need many resources. That will allow us to avoid any cycledef/dependency issues.

@CoryMartin-NOAA
Copy link
Contributor

Yes, @jiaruidong2017 can you close this PR and create a new one where the snow cover processing is done at the beginning of the snow analysis job?

@jiaruidong2017
Copy link
Contributor Author

@CoryMartin-NOAA Okay, I will proceed to work and create a new PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants