Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update snow obs preprocessing job #2946

Closed
Closed
Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
4098f10
Remove the GTS BUFR2IODA part of the snow obs prep job and
jiaruidong2017 Sep 9, 2024
8d1bc87
Change HOMEgfs to PARMgfs.
jiaruidong2017 Sep 10, 2024
6ff65bc
Merge branch 'develop' into feature/snow_gtsobs
jiaruidong2017 Sep 10, 2024
2295cab
Merge branch 'develop' into feature/snow_gtsobs
CoryMartin-NOAA Sep 11, 2024
5074d11
Made changes to stage the static GTS mapping files.
jiaruidong2017 Sep 12, 2024
f122edf
Change the name to be more descriptive.
jiaruidong2017 Sep 12, 2024
73b4ec0
Merge branch 'develop' into feature/snow_gtsobs
jiaruidong2017 Sep 15, 2024
086158d
Update IMS preprocessing job.
jiaruidong2017 Sep 15, 2024
2df23be
Rename IMS snow preprocessing jobs/names to be snowcover
jiaruidong2017 Sep 16, 2024
bb8775a
Merge branch 'develop' into feature/ims_snowcover
jiaruidong2017 Sep 17, 2024
5ddf552
Merge branch 'develop' into feature/ims_snowcover
jiaruidong2017 Sep 19, 2024
0bfbff7
Merge branch 'develop' into feature/ims_snowcover
jiaruidong2017 Sep 20, 2024
3c08705
Address reviewer's comments.
jiaruidong2017 Sep 27, 2024
850f672
Merge branch 'develop' into feature/ims_snowcover
jiaruidong2017 Oct 4, 2024
6f2fc66
Merge branch 'develop' into feature/ims_snowcover
jiaruidong2017 Oct 15, 2024
30c19d4
Merge branch 'feature/ims_snowcover' of https://github.com/jiaruidong…
jiaruidong2017 Oct 15, 2024
08bb8d8
Merge branch 'develop' into feature/ims_snowcover
jiaruidong2017 Oct 15, 2024
8a4a743
Merge branch 'develop' into feature/ims_snowcover
jiaruidong2017 Oct 22, 2024
b12a375
Update to use <taskvalid> for managing the rocoto dependency.
jiaruidong2017 Oct 23, 2024
679e57e
Fix the pynorms error.
jiaruidong2017 Oct 23, 2024
d97480a
Fix an error.
jiaruidong2017 Oct 23, 2024
ed78b8c
Update the name of dependency from prepsnowcover for ens recentering.
jiaruidong2017 Oct 23, 2024
eac7e1f
Update to run gdas_prepsnowcover job on every 00Z cycle.
jiaruidong2017 Oct 25, 2024
96254c4
Made changes as reviewer suggested.
jiaruidong2017 Oct 25, 2024
a00f9e8
Fix pynorm error.
jiaruidong2017 Oct 25, 2024
20b3b9c
Merge branch 'develop' into feature/ims_snowcover
DavidHuber-NOAA Oct 28, 2024
db3a912
Update the dependency for the enkfgdas_esnowrecen job.
jiaruidong2017 Oct 28, 2024
2e319a7
Merge branch 'feature/ims_snowcover' of https://github.com/jiaruidong…
jiaruidong2017 Oct 28, 2024
3d8c9d3
Update the esnowrecen job dependency.
jiaruidong2017 Oct 28, 2024
7148c4e
Update the gfs workflow for prepsnowcover job.
jiaruidong2017 Oct 30, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion env/HERA.env
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ if [[ "${step}" = "prep" ]] || [[ "${step}" = "prepbufr" ]]; then
export sys_tp="HERA"
export launcher_PREP="srun"

elif [[ "${step}" = "prepsnowobs" ]]; then
elif [[ "${step}" = "prepsnowcover" ]]; then

export APRUN_CALCFIMS="${APRUN_default}"

Expand Down
2 changes: 1 addition & 1 deletion env/HERCULES.env
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ case ${step} in
export sys_tp="HERCULES"
export launcher_PREP="srun"
;;
"prepsnowobs")
"prepsnowcover")

export APRUN_CALCFIMS="${APRUN_default}"
;;
Expand Down
2 changes: 1 addition & 1 deletion env/JET.env
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ if [[ "${step}" = "prep" ]] || [[ "${step}" = "prepbufr" ]]; then
export sys_tp="JET"
export launcher_PREP="srun"

elif [[ "${step}" = "prepsnowobs" ]]; then
elif [[ "${step}" = "prepsnowcover" ]]; then

export APRUN_CALCFIMS="${launcher} -n 1"

Expand Down
2 changes: 1 addition & 1 deletion env/ORION.env
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ if [[ "${step}" = "prep" ]] || [[ "${step}" = "prepbufr" ]]; then
export sys_tp="ORION"
export launcher_PREP="srun"

elif [[ "${step}" = "prepsnowobs" ]]; then
elif [[ "${step}" = "prepsnowcover" ]]; then

export APRUN_CALCFIMS="${launcher} -n 1"

Expand Down
2 changes: 1 addition & 1 deletion env/S4.env
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ if [[ "${step}" = "prep" ]] || [[ "${step}" = "prepbufr" ]]; then
export sys_tp="S4"
export launcher_PREP="srun"

elif [[ "${step}" = "prepsnowobs" ]]; then
elif [[ "${step}" = "prepsnowcover" ]]; then

export APRUN_CALCFIMS="${APRUN_default}"

Expand Down
2 changes: 1 addition & 1 deletion env/WCOSS2.env
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ if [[ "${step}" = "prep" ]] || [[ "${step}" = "prepbufr" ]]; then
export sys_tp="wcoss2"
export launcher_PREP="mpiexec"

elif [[ "${step}" = "prepsnowobs" ]]; then
elif [[ "${step}" = "prepsnowcover" ]]; then

export APRUN_CALCFIMS="${APRUN_default}"

Expand Down
4 changes: 2 additions & 2 deletions jobs/JGLOBAL_PREP_SNOW_OBS → jobs/JGLOBAL_PREP_SNOWCOVER
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

source "${HOMEgfs}/ush/preamble.sh"
export DATA=${DATA:-${DATAROOT}/${RUN}snowanl_${cyc}}
source "${HOMEgfs}/ush/jjob_header.sh" -e "prepsnowobs" -c "base prepsnowobs"
source "${HOMEgfs}/ush/jjob_header.sh" -e "prepsnowcover" -c "base prepsnowcover"

##############################################
# Set variables used in the script
Expand All @@ -25,7 +25,7 @@ RUN=${GDUMP} YMD=${gPDY} HH=${gcyc} declare_from_tmpl -rx \

###############################################################
# Run relevant script
EXSCRIPT=${GDASSNOWPREPPY:-${SCRgfs}/exglobal_prep_snow_obs.py}
EXSCRIPT=${GDASSNOWPREPPY:-${SCRgfs}/exglobal_prep_snowcover.py}
${EXSCRIPT}
status=$?
[[ ${status} -ne 0 ]] && (echo "FATAL ERROR: Error executing ${EXSCRIPT}, ABORT!"; exit "${status}")
Expand Down
4 changes: 2 additions & 2 deletions jobs/rocoto/prepsnowobs.sh → jobs/rocoto/prepsnowcover.sh
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ source "${HOMEgfs}/ush/preamble.sh"
status=$?
[[ ${status} -ne 0 ]] && exit "${status}"

export job="prepsnowobs"
export job="prepsnowcover"
export jobid="${job}.$$"

###############################################################
Expand All @@ -21,6 +21,6 @@ export PYTHONPATH

###############################################################
# Execute the JJOB
"${HOMEgfs}/jobs/JGLOBAL_PREP_SNOW_OBS"
"${HOMEgfs}/jobs/JGLOBAL_PREP_SNOWCOVER"
status=$?
exit "${status}"
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
#! /usr/bin/env bash

########## config.prepsnowobs ##########
########## config.prepsnowcover ##########
# Snow Obs Prep specific

echo "BEGIN: config.prepsnowobs"
echo "BEGIN: config.prepsnowcover"

# Get task specific resources
. "${EXPDIR}/config.resources" prepsnowobs
. "${EXPDIR}/config.resources" prepsnowcover

export IMS_OBS_LIST="${PARMgfs}/gdas/snow/prep/prep_ims.yaml.j2"

Expand All @@ -15,4 +15,4 @@ export FIMS_NML_TMPL="${PARMgfs}/gdas/snow/prep/fims.nml.j2"

export IMS2IODACONV="${USHgfs}/imsfv3_scf2ioda.py"

echo "END: config.prepsnowobs"
echo "END: config.prepsnowcover"
4 changes: 2 additions & 2 deletions parm/config/gfs/config.resources
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ if (( $# != 1 )); then
echo "Must specify an input task argument to set resource variables!"
echo "argument can be any one of the following:"
echo "stage_ic aerosol_init"
echo "prep prepsnowobs prepatmiodaobs"
echo "prep prepsnowcover prepatmiodaobs"
echo "atmanlinit atmanlvar atmanlfv3inc atmanlfinal"
echo "atmensanlinit atmensanlobs atmensanlsol atmensanlletkf atmensanlfv3inc atmensanlfinal"
echo "snowanl esnowrecen"
Expand Down Expand Up @@ -151,7 +151,7 @@ case ${step} in
memory="40GB"
;;

"prepsnowobs")
"prepsnowcover")
walltime="00:05:00"
ntasks=1
threads_per_task=1
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
#!/usr/bin/env python3
# exglobal_prep_snow_obs.py
# exglobal_prep_snowcover.py
# This script creates a SnowAnalysis object
# and runs the prepare_GTS and prepare_IMS method
# which perform the pre-processing for GTS and IMS data
# and runs the prepare_IMS method which perform
# the pre-processing for IMS data
import os

from wxflow import Logger, cast_strdict_as_dtypedict
Expand All @@ -20,5 +20,4 @@

# Instantiate the snow prepare task
SnowAnl = SnowAnalysis(config)
if SnowAnl.task_config.cyc == 0:
SnowAnl.prepare_IMS()
SnowAnl.prepare_IMS()
4 changes: 2 additions & 2 deletions workflow/applications/gfs_cycled.py
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,7 @@ def _get_app_configs(self):
configs += ['prepobsaero']

if self.do_jedisnowda:
configs += ['prepsnowobs', 'snowanl']
configs += ['prepsnowcover', 'snowanl']
if self.do_hybvar:
configs += ['esnowrecen']

Expand Down Expand Up @@ -156,7 +156,7 @@ def get_task_names(self):
gdas_gfs_common_tasks_before_fcst += ['sfcanl', 'analcalc']

if self.do_jedisnowda:
gdas_gfs_common_tasks_before_fcst += ['prepsnowobs', 'snowanl']
gdas_gfs_common_tasks_before_fcst += ['prepsnowcover', 'snowanl']

wave_prep_tasks = ['waveinit', 'waveprep']
wave_bndpnt_tasks = ['wavepostbndpnt', 'wavepostbndpntbll']
Expand Down
11 changes: 11 additions & 0 deletions workflow/rocoto/gfs_cycled_xml.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,17 @@ def get_cycledefs(self):
sdate_str = sdate.strftime("%Y%m%d%H%M")
strings.append(f'\t<cycledef group="gdas">{sdate_str} {edate_str} {interval_str}</cycledef>')

if self._app_config.do_jedisnowda:
sdate_snocvr = self._base['SDATE']
edate_snocvr = self._base['EDATE']
interval_snocvr = to_timedelta(f"24:00:00H")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
interval_snocvr = to_timedelta(f"24:00:00H")
interval_snocvr = to_timedelta('24H')

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Made changes. Thanks.

sdate_snocvr = sdate_snocvr + interval_snocvr
sdate_snocvr_str = sdate_snocvr.replace(hour=0, minute=0, second=0).strftime("%Y%m%d%H%M")
edate_snocvr_str = edate_snocvr.strftime("%Y%m%d%H%M")
interval_str = timedelta_to_HMS(interval_snocvr)
if sdate_snocvr <= edate_snocvr:
DavidHuber-NOAA marked this conversation as resolved.
Show resolved Hide resolved
strings.append(f'\t<cycledef group="gdas_prep_snocvr">{sdate_snocvr_str} {edate_snocvr_str} {interval_str}</cycledef>')

if self._app_config.gfs_cyc != 0:
sdate_gfs = self._base['SDATE_GFS']
edate_gfs = self._base['EDATE_GFS']
Expand Down
18 changes: 11 additions & 7 deletions workflow/rocoto/gfs_tasks.py
Original file line number Diff line number Diff line change
Expand Up @@ -564,21 +564,22 @@ def aeroanlfinal(self):

return task

def prepsnowobs(self):
def prepsnowcover(self):

deps = []
dep_dict = {'type': 'task', 'name': f'{self.run}prep'}
deps.append(rocoto.add_dependency(dep_dict))
dependencies = rocoto.create_dependency(dep=deps)

resources = self.get_resource('prepsnowobs')
task_name = f'{self.run}prepsnowobs'
cycledef = 'gdas_prep_snocvr'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
cycledef = 'gdas_prep_snocvr'

resources = self.get_resource('prepsnowcover')
task_name = f'{self.run}prepsnowcover'
task_dict = {'task_name': task_name,
'resources': resources,
'dependency': dependencies,
'envars': self.envars,
'cycledef': self.run.replace('enkf', ''),
'command': f'{self.HOMEgfs}/jobs/rocoto/prepsnowobs.sh',
'cycledef': cycledef,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
'cycledef': cycledef,
'cycledef': 'gdas_prep_snocvr',

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Made changes as suggested. Thanks.

'command': f'{self.HOMEgfs}/jobs/rocoto/prepsnowcover.sh',
'job_name': f'{self.pslot}_{task_name}_@H',
'log': f'{self.rotdir}/logs/@Y@m@d@H/{task_name}.log',
'maxtries': '&MAXTRIES;'
Expand All @@ -591,7 +592,10 @@ def prepsnowobs(self):
def snowanl(self):

deps = []
dep_dict = {'type': 'task', 'name': f'{self.run}prepsnowobs'}
if f'@H' == '00':
WalterKolczynski-NOAA marked this conversation as resolved.
Show resolved Hide resolved
dep_dict = {'type': 'task', 'name': f'{self.run}prepsnowcover'}
else:
dep_dict = {'type': 'task', 'name': f'{self.run}prep'}
deps.append(rocoto.add_dependency(dep_dict))
dependencies = rocoto.create_dependency(dep=deps)

Expand All @@ -614,7 +618,7 @@ def snowanl(self):
def esnowrecen(self):

deps = []
dep_dict = {'type': 'task', 'name': f'{self.run.replace("enkf","")}prepsnowobs'}
dep_dict = {'type': 'task', 'name': f'{self.run.replace("enkf","")}prepsnowcover'}
KateFriedman-NOAA marked this conversation as resolved.
Show resolved Hide resolved
deps.append(rocoto.add_dependency(dep_dict))
dep_dict = {'type': 'task', 'name': f'{self.run.replace("enkf","")}snowanl'}
deps.append(rocoto.add_dependency(dep_dict))
Expand Down
2 changes: 1 addition & 1 deletion workflow/rocoto/tasks.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ class Tasks:
'eobs', 'eomg', 'epos', 'esfc', 'eupd',
'atmensanlinit', 'atmensanlobs', 'atmensanlsol', 'atmensanlletkf', 'atmensanlfv3inc', 'atmensanlfinal',
'aeroanlinit', 'aeroanlvar', 'aeroanlfinal', 'aeroanlgenb',
'prepsnowobs', 'snowanl', 'esnowrecen',
'prepsnowcover', 'snowanl', 'esnowrecen',
'fcst',
'atmanlupp', 'atmanlprod', 'atmupp', 'goesupp',
'atmos_prod', 'ocean_prod', 'ice_prod',
Expand Down
Loading