POD's access to the multirun cases- How?

[aradhakrishnanGFDL]

This discussion stems from the multirun implementation issue #309 and a related PR with an example POD #313, where we begin to discuss how the dictionary hand off to the POD may happen. This discussion is to welcome all ideas and assess the cost benefit in terms of implementation, backward compatibility and value to POD developers, etc. Since the discussion is tied to the initial prototype, more than one way could be explored possibly to arrive at a stable approach, considering the big picture.

[aradhakrishnanGFDL]

Gathering ideas..

Quoting @wrongkindofdoctor from #313

#313 is a good reference, since it make sense to make the case list available to the POD to parse from jsonc file (could be a separate flle, or part of POD's settings.jsonc file). The framework launches PODs as a subprocess, and specifies the environment variables using the env=env_vars option. Subprocess does allow passing arguments to programs that it calls, but it does not seem like it can handle the nested dictionary structure we're working with.

The only way to pass the case_list dictionary from the framework to the POD would be with multiprocessing. This method is more complex, and carries the risk of overloading system resources and race conditions.

Whether the case_list comes from the framework or directly from a jsonc file (or yaml), the POD would need a method to receive case_list. Users could obtain the case_list from a file using the read_json routine in src/util/filesystem.py:

from src.util import read_json
code_root = "/home/mdtf/MDTF-diagnostics"
file_name = "src/default_tests.jsonc"
json_obj = read_json(os.path.join(code_root, file_name)'
case_list = json_obj.get('case_list')

Using multiprocessing:

Framework routine following the example in the multprocessing link
(not entirely sure if this is correct)
import multiprocessing
p = multiprocessing.Process(target=example.py, args=(case_list))
p.start()
p.join()

POD case_list intake routine
def read_case_list(**case_list):
access each dictionary element
for k, v in case_list.items():
#display case name
print(k)
for k2 in v:
# case name attribute keys and values
print(k2,v[k2])

[aradhakrishnanGFDL]

Some thoughts..

1- Accessing the dictionary somewhat directly from a POD seems more usable.

A) Saving out the caselist dictionary to a separate file, like what @wrongkindofdoctor suggested or simply have a caselist.py written out let us use the dictionary in a POD more directly.

e.g.
framework writes out caselist.py
The POD code would be something like this

B) If the framework has a utility to return the dictionary in a self contained way, then the python PODs can also use it directly without writing out a file.

E.g.

Here is how the POD "may" look like..
And filesystem.py (or something else) may look like

2- We should probably also have the input file pointers (presently env variable TAS_FILE) from the different cases passed along in the same dictionary? If yes, this may also help us with a couple of things down the line a) recording provenance for a POD run b) extending the preprocessing feature for performance to rely on this for cached or intermediate preprocessed output.

3- This doc also suggests that there is a catalog (intake style, csv) which drives the preprocessing. If we could persist this catalog and (enhance it later) if-needed, this may be an easier alternative too.

E.g. framework persists the catalog, framework provides helper utility to parse catalog and provide CASENAME, TAS_FILE, POD has direct access to the catalog for xarray and intake-esm enthusiasts, POD also has access to the catalog via the helper utility.
Not to miss that all of these things could potentially help in some way with guiding a user to generate config files for larger ensembles/cases, etc.

The cons is that providing more than one way is a bit dangerous, but not unreasonable at an early stage. How all this works for a POD not written in python should also be considered.

(note- for another discussion, I see a --disable-preprocessor option in the docs, so we should probably check it out if not already to see if there may be easier ways to implement [4] to determine cost-benefit )

thoughts? comments? other ideas?

[jkrasting]

Thanks @aradhakrishnanGFDL for starting this discussion and @wrongkindofdoctor for the prompts!

I think a direct python-to-python hand-off (via multiprocessing for example) has some logistical/performance issues as @wrongkindofdoctor points out. Creating a separate caselist.py file (or alternatively a pickle file) also only addresses the Python angle - we have NCL to contend with.

At the end of the preprocessing, if the framework dumped the metadata dictionary into a JSON file in the $WK_DIR, it's trivial to load that in via json.load(). If we wanted to, we could even make a little function or utility to handle this for the POD developers. Similarly for NCL, we could have a stand-alone tool that read the JSON file and delivered the info to the NCL PODs.

[aradhakrishnanGFDL]

Thanks @jkrasting- Right, using multiprocessing calls to pass the dictionary into the POD may not be the most usable way, indeed complex as @wrongkindofdoctor says.

Thinking in terms of the broader workflow as well, a separate file (py, csv, json..) is plausible for reasons I mentioned earlier. @bitterbark had mentioned the NCL scripts all use a python driver script, and @bitterbark also plans to have a dict-to-ncl-compatible converter as well. (thinking ahead

I still think evaluating what the data catalog looks like today would be helpful for planning purposes.

Does $WK_DIR always get wiped off at the end of the run?

[aradhakrishnanGFDL]

The data catalog appears to be a dynamic one (as mentioned in the doc). So, this means there is indeed an esm_datastore that is heavily relied on for pre-processing, which is nice. But, from what I can tell it seems to hold info regarding the input datasets found in say MODEL_DATA_ROOT, which is then pre-processed, filtered..to get our final cleaned up datasets, etc. Since the latter is not in the catalog, this option to me is ruled out for the multicase/multirun implementation.

[bitterbark]

Regarding the catalog: my perception is that it is currently only used to preprocess data, by reading in the native variables and writing them out as prescribed by the framework (and cutting the time range as necessary). If there is a way to use it to pass data directly to the PODs (even if an interface is required to the NCL code), it seems this would be a good, eventual, use of the code (but I don't think we're near that now).

[andrewgettelman]

I thought reading the object out of the json file looked simpler if there are potential issues with multiprocessing.....

[wrongkindofdoctor]

@aradhakrishnanGFDL There is a "keep_temp" option to retain temporary directories in the WK_DIR that is set to .false. by default. Note that it is listed in src/cli_template.jsonc, and sites/NOAA_GFDL/cli_gfdl.jsonc, which are used to set cli default values, but not in default_tests.jsonc.

[aradhakrishnanGFDL]

@wrongkindofdoctor I do not think keep_temp overriding works in my preliminary tests. This may be out of scope for this discussion, tabling it for an issue later.

[aradhakrishnanGFDL]

If there is a python-driver available for NCL, json is probably still readily usable there. (If not, we could consider csv because NCL seems to have read methods for csv.) If we do have an R application in the future, json still works. (so no to pickle perhaps)

case_settings.json (further below) for @jkrasting example POD seems to kinda fit in.
Note that WK_DIR is another key that will probably be needed for the #313 to run to completion. (This means #313 will need some minor code mods)

In the POD (something like the following, or fancier)

with open('case_settings.json', "r") as outfile:
        case_info = json.load(outfile)

case_settings.json (dummy name for now)

{
  "CASENAME": {
    "NAME": "CMIP_Synthetic_r1i1p1f1_gr1_00010101-00101231",
    "TAS_FILE": "/Users/ar46/mdtf/forks/jpk/wkdir/MDTF_CMIP_Synthetic_r1i1p1f1_gr1_00010101-00101231_1_10.v14/example_multicase/day/CMIP_Synthetic_r1i1p1f1_gr1_00010101-00101231.tas.day.nc",
    "tas_var": "tas",
    "time_coord": "time",
    "lon_coord": "lon"
  },
  "CASENAME1": {
    "NAME": "CMIP_Synthetic_r1i1p1f1_gr1_00010101-00101231",
    "TAS_FILE": "/Users/ar46/mdtf/forks/jpk/wkdir/MDTF_CMIP_Synthetic_r1i1p1f1_gr1_00010101-00101231_1_10.v14/example_multicase/day/CMIP_Synthetic_r1i1p1f1_gr1_00010101-00101231.tas.day.nc",
    "tas_var": "tas",
    "time_coord": "time",
    "lon_coord": "lon"
  },
  "WK_DIR": "/Users/ar46/mdtf/forks/jpk/wkdir/MDTF_CMIP_Synthetic_r1i1p1f1_gr1_00010101-00101231_1_10.v14/"
}

@wrongkindofdoctor and everyone- Thoughts?

[aradhakrishnanGFDL]

Perhaps the POD settings json @wrongkindofdoctor referred to earlier already has access to some of these info, so some of the things in my sample case_settings.json may be redundant. But, in general what else at a high-level should get added that pertains to the cases? start/end year? what other info is needed?

(p.s config_save.json in the outdir may be relevant for dev )

[bitterbark]

I suggest we use CASENAME2 instead of CASENAME1 because the latter is ambiguous in this case:
If we didn't need backward compatibility, we would use CASENAME1, CASENAME2, CASENAME3,...

If we set CASENAME1 = CASENAME as a redundancy, then pods that can only use one case have complete backward compatibility (CASENAME) while pods that use multiple cases can have a consistent list (CASENAME1, CASENAME2, CASENAME3,)

[aradhakrishnanGFDL]

@bitterbark yeah, CASENAME1, CASENAME2, CASENAME3 does sound neat with the backward compatibility special conditional assignment. @wrongkindofdoctor Do you see any problems with this approach?

[wrongkindofdoctor]

This approach seems reasonable. In fact, it won't (or shouldn't) matter what you name each case (as long as each name is unique), since the intent is to have the framework loop through the case dictionary parsed by the jsonc routine similarly to the way I've modified the framework to loop through the caselist elements after reading in the multirun_config_template.jsonc.

I don't have any more advice on what this part of the implementation will look like, as I've only made it to the preprocessor step, and am working to assign varlist attributes to each case. Each varlist contains the POD variable info specified in settings.jsonc, plus other info that includes the [VAR_NAME]_file (e.g., TAS_FILE for the example POD), and the path to the file associated with each variable/to each case for the pods to work on that are assigned by the framework (if the user has not specified them). Right now, the framework attaches one varlist to a POD object that is passed to POD run step.

[bitterbark]

Regarding the NCL pods: I think we should be able to call a json reader from the python interface scripts
an example here
although this isn't currently done.

I don't believe that the NCL pods currently read the settings.json files. The framework reads them (at the preprocessing stage?) and the only part that I have used within the pod code is the env var list. I'm not sure where this comes from but it seems the framework has to be parsing it into env vars that are then available in NCL.

[bitterbark]

I don't believe we should spend too much time on making fancy solutions for NCL as it should be getting phased out. At this point we want the easiest solution that will function in NCL and is optimal for python.

[bitterbark]

Regarding the meeting conversation about changing output directory structure:
The most important thing to POD developers is clear and consistent interfaces to the framework, both for input and for output. For example, if we change the precedence of directories from case to pod, WKDIR (if that's what the env var is currently called) should be updated in such a way that the POD developers don't need to change their code. Another example is how the env var that holds path to the input file (which is postprocessed) is staying the same (with the extension of case numbers).

[aradhakrishnanGFDL]

To summarize meeting notes related to this topic:

1. Framework must write a readable, well-formatted, indented case-settings.json to wkdir (output paths related topics may arise later after the prototype, but defining the top-level directories for POD to expect input or output is beneficial future looking changes)

2. Framework adds a helper utility function (leveraging existing code base) that returns the case information that the POD needs, through either a python dictionary or a custom utility to convert the dictionary to environment variables. The former is prioritized for the prototype. The latter (dict to env) will be revisited with a NCL POD use-case next.

New example POD will probably reflect better usage of this.

Dummy example call in POD:

from jsontodict import get_json_dict
case_info = get_json_dict(..)
# (where case_info is the dictionary) 

#POD code continues 
for k, v in case_info.items():
   print(k,v)
   print(case_info[k]["TAS_FILE"])

3. The helper utility abstraction will help us assess if intake-esm data catalog could be substituted here for PODs to access the catalogs directly, down the line. This is due to the fact that the framework uses esm.datastore for preprocessing already, so reusing those code bits and enhancing them for "post-processed" data will help users readily use these catalogs in their PODs (especially if one uses xarray) as well.

(@wrongkindofdoctor @bitterbark @jkrasting Please edit this directly to capture meeting outcome accurately. Thanks )

P.S- This is being updated based on discussion thread.

[bitterbark]

I don't see how to edit directly, but I would modify (2) as:
2. Framework adds a helper utility function in utils.py that returns the case information that the POD needs, through either a python dictionary or converting the dictionary to environment variables

[wrongkindofdoctor]

We may be able to leverage exisiting json utilities for reading, writing, and formatting json output in src/util/filesystem.py

[aradhakrishnanGFDL]

thanks! Yeah, my mind was thinking google doc when I typed it, edit is not possible by others, right.

Just to clarify, "converting the dictionary to environment variables" dict to env variables is for NCL POD compatibility for multi-case? or something else?

(The env variable approach for a single case should still be available from MDTF for backward compatibility, per design discussion )

[bitterbark]

Yes, just for multi case for NCL pods.

…

On Feb 3, 2022, at 8:06 AM, Aparna Radhakrishnan ***@***.***> wrote:  thanks! Yeah, my mind was thinking google doc when I typed it, edit is not possible by others, right. Just to clarify, "converting the dictionary to environment variables" dict to env variables is for NCL POD compatibility for multi-case? or something else? (The env variable approach for a single case should still be available from MDTF for backward compatibility, per design discussion ) — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you were mentioned.

[aradhakrishnanGFDL]

Ok, I have updated the original steps. Just wanted to make sure I capture the priority for the prototype (with the example POD). Let me know if I need to tweak the language, please.