Add v1 data #46
Add v1 data #46
Conversation
There was a problem hiding this comment.
Porting v1 data from https://e3sm.org/data/ to https://docs.e3sm.org/e3sm_data_docs/_build/html/index.html is low priority at the moment. Notably, many paths (HPSS paths, ESGF links) may be updated in the near future, so it doesn't really make sense to make additions now that will just need editing later.
I'm therefore pushing the work I have so far to GitHub, with notes to make it easier to return to development later on.
The current implementation organizes data as version > campaign/group > simulation vs reproduction > table of data. Example screenshot of https://portal.nersc.gov/project/e3sm/forsyth/data_docs/html/v1/WaterCycle/simulation_data/simulation_table.html:
| # Native (experiment, and optionally model_version) | ||
| # CMIP6 (activity_id, source_id, experiment_id) | ||
|
|
||
| # TODO: figure out if we're returning the correct things |
There was a problem hiding this comment.
I've listed the numbers of items returned in the comments below, for e3sm.org vs the current implementation in this PR. As the table below reveals, the number of returned items don't always match up as expected.
| # amip | 527 | 30 | 527 | 30 | 3 rows on data_docs | ||
| # amip_1850_allF | x | 30 | | ||
| # amip_1850_aeroF | x | 30 | | ||
| # TODO: only return Native for some experiments... |
There was a problem hiding this comment.
https://e3sm.org/data/get-e3sm-data/released-e3sm-data/v1-1-deg-data-cmip6/ shows CMIP6 and Native for piControl, historical, 1pctCO2, abrupt-4xCO2, amip, but only Native for abrupt-4xCO2-ext300yrs, amip_1850_allF, amip_1850_aeroF.
| esgf = f"`Native <https://esgf-node.llnl.gov/search/e3sm/?model_version={model_version[1:]}_0&experiment={experiment}&ensemble_member=ens{ensemble_num}>`_" | ||
| # Note that `[1:]`` removes `v` from `model_version` | ||
| # | ||
| # Query parameters: |
There was a problem hiding this comment.
The query parameters for the ESGF links differ between v1 and v2.
|
|
||
| hpss_path = f"/home/projects/e3sm/www/{self.group}/E3SM{self.model_version}/{self.resolution}/{self.simulation_name}" | ||
| if "hpss_path" in simulation_dict: | ||
| hpss_path = simulation_dict["hpss_path"] |
There was a problem hiding this comment.
If the HPSS path doesn't match our expected form, we're going to just have to get it directly from the csv.
|
This is WIP. Tony is working on transfer the v1 publication directory from LLNL to NERSC HPSS. Also some v1 simulations need to be copied from their original path to /home/projects/e3sm/www |
Add v1 data. Resolves #43