HAFS moving nest not working in Derecho

[murad-me]

Hello,

I can run the HAFS-A model without nesting successfully on the NCAR Derecho system, both with and without the ocean model (MOM6). However, when running HAFS A with a moving nest, my simulation crashes with the following SIGTERM error:

I am attaching hfsa.conf, hafs_forecast.log, hafs_atm_prep.log and hafs_atm_prep_mvnest.log for reference. I would appreciate it if anyone could help me with this.

Thanks in advance,
Murad
hfsa.conf.txt
hafs_atm_prep.log.txt
hafs_atm_prep_mvnest.log.txt
hafs_forecast.log.txt

[muradme]

Hi everyone,

I wanted to follow up on the discussion I started. I haven’t received a response yet and was wondering if there is any suggestions or further insights available on this topic.

Any updates or additional information would be greatly appreciated.

Best regards,
Murad

[jdkublnick]

Hello, Muradme! We have not forgotten about your question; SMEs were contacted last week regarding your question. I will reach out with more information as soon as I have it. Thank you, and have a great day!

[muradme]

Thank you @jdkublnick, I greatly appreciate you contacting SME and giving me an update about this.

I want to add that I also have found this range_check error in the above-attached forecast log (line 27926).

---

PASS: fcstRUN phase 1, n_atmsteps = 1 time is 0.975311
At forecast hour 0.050 mean abs pgr change is 15.72907360 hPa/hr
max abs change 0.0637798200 bar at tile=1 i=659 j=631
pgr at that point 0.8162924461 bar lat= 16.739845 lon= 267.416629
Range_check Warning:TA max = 364.5764 min = 189.3950
Range violation on: 2024/ 9/25 18: 3: 0
NOTE from PE 1920: ==> Error from range_check: data out of bound
Warn_K= 81 (i,j)= 660 615 (lon,lat)=267.396 17.023 TA = 364.57642
K= 80 300.21954
Warn_K= 81 (i,j)= 660 616 (lon,lat)=267.396 17.005 TA = 363.09659
K= 80 296.06705
NOTE from PE 1920: ==> Error from range_check: data out of bound

However, I don't know if it's relevant to this SIGTERM error or not. I have run two other cases, which did not have this range_check error, but still crashed with SIGTERM error at a similar point.

Best regards,
Murad

[gspetro-NOAA]

Hi @murad-me ,

We discussed this issue with the developers yesterday, and unfortunately, they don't have access to Derecho to troubleshoot and generally are not responsible for providing community support. Although our team provides community support for certain repositories on supported platforms, Derecho is not yet a supported platform for HAFS, and we do not currently have resources on our team to assist with this troubleshooting. We will leave the issue open in case there are community members working with HAFS on Derecho who are able to assist, now or in the future. I'm sorry we can't provide a more helpful answer at this time. We did forward your request (Issue #278 ) to management for consideration, but currently, there are no plans to port HAFS to Derecho. I am hopeful that that could change in the future, but it likely wouldn't be soon. :-/

Best,
Gillian Petro

[gspetro-NOAA]

Hi @murad-me ,

I did discover one potentially useful piece of information. The UFS Weather Model developers maintain regression testing baselines at /glade/derecho/scratch/epicufsrt/ufs-weather-model/RT/NEMSfv3gfs/develop-20241121. There are also a number of HAFS-related regression tests that are run regularly on UFS WM supported systems. I don't think the HAFS-related RT configurations are run on Derecho, but you may be able to glean some information from both the hafs_* RTs and other RTs that are run on Derecho. You can take a look at the tests here: https://github.com/ufs-community/ufs-weather-model/tree/develop/tests/tests. You can also check out the UFS WM modulefiles for Derecho and the default_vars.sh script, which contains many of the default settings (including some machine-specific ones) for different sets of tests. The User's Guide has more info on running RTs and some background on the RTs. I'm hoping this can help you on your troubleshooting journey!

Best,
Gillian

[muradme]

Hi @gspetro-NOAA,

Thanks for your reply. I have got some valuable insights about the environments for HAFS in derecho. I have used this environment, especially spack-stack v1.6 (I used spack-stack 1.8 and some of my installed libraries before), but the current error persists. I am curious if there is any debug mode to check nesting issue. I could run HAFS-A with atm and atm-ocn coupled mode, and without nesting it works. Just when I apply moving nest, it crashes with SIGTERM error. If there was debug flag in namelist/confing files or somewhere else for moving nest, I could understand what is going on. Additionally, can you suggest me any potential part of HAFS code to check for moving nest, as I am facing issues only for moving nest?

Thanks a lot,
Murad