Add multirun capability to blocking_neale POD

[bitterbark]

Description
blocking_neale POD can now run with multiple input model cases
Html page is written dynamically for however many cases are present
Modifications made to make date selection for each case and each observation more flexible (see diagnostics/blocking_neale/settings.jsonc)

Notes:

• This has an example of an interactive web page write, which is necessary
  since we don't know how many cases will be present. However, at this stage it
  is written into the POD_DIR instead of the
  WK_DIR. This is not a good long-term solution because users won't always
  have write access to their code base.

(This is necessary because the framework complains if there isn't a $POD_NAME.html file in the $POD_DIR.

  File "/glade/u/home/bundy/mdtf/MDTF_3_main/MDTF-diagnostics/src/output_manager.py", line 127, in make_pod_html
    raise util.MDTFFileNotFoundError(test_path)
src.util.exceptions.MDTFFileNotFoundError: [Errno 2] No such file or  directory:
'/glade/u/home/bundy/mdtf/MDTF_3_main/MDTF-diagnostics/diagnostics/blocking_neale/blocking_neale.html'

If I leave a stub file instead, it overwrites the dynamic file. So I've reverted to writing the dynamic webpage into the POD_DIR but we should work on this in the future

• For the multi-run, PODs are the top level of organization (single level =
  model case)
  1. The top level MDTF index.html with a link to all pods is in (each)? pod's
     dir. I haven't run multiple PODS in the multicase run to see how this works
     (are all links in all pods)? but if this hasn't been done we should try it!
  2. This changes how the tar files are generated. Single case tar files are
     made of the top level dir, multi are the individual pod dirs. Again, I'm
     not sure if this has been tested on multiple pods in a multi case scenerio.

How Has This Been Tested?
Ran in both multi-run and single-run mode using my own input files. Framework reports success and examination of website agrees.

Multi:
https://www2.cgd.ucar.edu/cms/bundy/Projects/diagnostics/mdtf/mdtf_figures/blocking_2023/20230810/multi/blocking_neale.html
Generated with

"pod_list" : [
"blocking_neale"
// "example_multicase"
],
// Each CASENAME corresponds to a different simulation/output dataset
"case_list" : [
{
"CASENAME" : "cesm_mdtfv3_timeslice",
"model" : "CESM",
"convention" : "CESM",
"FIRSTYR" : 2013,
"LASTYR" : 2013
},
{
"CASENAME" : "QBOi.EXP1.AMIP.001",
"model" : "CESM",
"convention" : "CESM",
"FIRSTYR" : 1979,
"LASTYR" : 1981
}

Single:
https://www2.cgd.ucar.edu/cms/bundy/Projects/diagnostics/mdtf/mdtf_figures/blocking_2023/20230810/single/blocking_neale/blocking_neale.html

Generated with:
"case_list" : [
{
"CASENAME" : "cesm_mdtfv3_timeslice",
"model" : "CESM",
"convention" : "CESM",
"FIRSTYR" : 2013,
"LASTYR" : 2013,
"pod_list": [
// Optional: PODs to run for this model only (defaults to all)
"blocking_neale"

Checklist:

• [ X] My branch is up-to-date with the NOAA-GFDL main branch, and all merge conflicts are resolved
• [ X] The scripts are written in Python 3.10 or above (preferred; required if funded by a CPO grant), NCL, or R
• [ X] All of my scripts are in the diagnostics/[POD short name] subdirectory, and include a main_driver script, template html, and settings.jsonc file
• [ O] I have made corresponding changes to the documentation in the POD's doc/ subdirectory
  will do immediately following, but I don't want to lose all of this PR doc by trying to commit that!
• [x ] I have requested that the framework developers add packages required by my POD to the python3, NCL, or R environment yaml file if necessary, and my environment builds with conda_env_setup.sh
• [ O] I have added any necessary data to input_data/obs_data/[pod short name] and/or input_data/model/[pod short name]
  Data is available on globus
• [X ] My code is portable; it uses MDTF environment variables, and does not contain hard-coded file or directory paths
• [ n/a] I have provided the code to generate digested data files from raw data files
• [ na/] Each digested data file generated by the script contains numerical data (no figures), and is 3 GB or less in size
• [ X] I have included copies of the figures generated by the POD in the pull request
  Single run
  Multi run
• [ X] The repository contains no extra test scripts or data files

[bitterbark]

I'm hoping that the failed tests are related to @wrongkindofdoctor's PRs this afternoon about fixing some failing tests.

Also, I'm not sure how my fork ended up with these commits by Tom from 4 years ago? Let me know if you want me to clean it up and try again.

[wrongkindofdoctor]

@bitterbark Yes, the failures are from ci-related issues. I put in a couple of fixes, so you can verify that your code updates pass the ubuntu build tests once you pull them in.

Your branch is derived from the old master branch, so that is probably why you have some lingering commits from Tom. Don't worry about cleaning up this PR, but checking out a new development branches from the main branch in your fork in the future should resolve this (i.e., every PR you submit should correspond to a unique branch).

[wrongkindofdoctor]

@bitterbark Overall, the modifications look good. I think that customizing the blocking_neale html template was a solid choice for the time being, and I appreciate the effort to delve into the code "guts". I will use the html file as a reference when I refactor the output manager. I just need you to sync your master branch with the main branch so that just the ubuntu tests run on your changes.

[bitterbark]

When I synced on the github web interface, it says I have merge conflicts that need to be resolved. However, on this PR I see it says "This branch has no conflicts with the base branch." And it looks like you just merged it. So is there something else I need to do?

[wrongkindofdoctor]

@bitterbark Your main branch should be alright if you haven't modified it. The master branch probably has conflicts because git can't resolve the updates with your commit history. If you don't have the option to resolve the conflicts with the web interface, you can update your remote and local main branches, then merge the main branch into your local master branch (assuming that's the branch you're trying to update). Resolve the merge conflicts in the problem files, commit the changes, and push them to your remote master branch.

[bitterbark]

Thanks. I switched my main branch to be the default and renamed master to master-don't-use (for myself to remember). Can I just leave it alone and carryon with my main branch from now on?

…

On Thu, Aug 17, 2023 at 9:30 AM Jess ***@***.***> wrote: Your main branch should be alright if you haven't modified it. The master branch probably has conflicts because git can'r resolve the updates with your commit history (I don't know why this is). If you don't have the option to resolve the conflicts with the web interface, you can update your remote and local main branches, then merge the main branch into your local master branch (assuming that's the branch you're trying to update). Resolve the merge conflicts in the problem files, commit the changes, and push them to your remote master branch. — Reply to this email directly, view it on GitHub <#472 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHLJVQCUCW4DBPM63YR7WVDXVY2JRANCNFSM6AAAAAA3LV6IIY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- My working day may not be the same as your working day. Please do not feel obliged to reply to this email outside of your normal hours.

[wrongkindofdoctor]

@bitterbark Yes. Keep your copy of the main branch clean, and check out development branches from main moving forward.