Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Path updates #98

Open
orianac opened this issue Feb 3, 2022 · 1 comment
Open

Path updates #98

orianac opened this issue Feb 3, 2022 · 1 comment

Comments

@orianac
Copy link
Member

orianac commented Feb 3, 2022

  1. We have the same target file (make_coarse_obs_path) for both get_coarse_obs_task and return_coarse_obs_full_time_task in bcsd_flow.py. I don't think this is right, unless you're intentionally trying to overwrite? But wouldn't prefect instead just always think that the task is cached because the target already exists by the time it gets to that task? Perhaps there is some special magic though that I don't see? (Update: adding the grid_spec and chunking_method back into the obs_identifier should help with this)

  2. Right now the pyramid files are stored with a path schema that looks like this:
    'az://flow-outputs/results_testing/pyramid_{daily,monthly,annual}/MIROC6/ssp370/tasmax/40.0_50.0_-125.0_-115.0/1981_2010/1981_2099/.pyr'.

We settled on changing that to: 'az://flow-outputs/results_testing/pyramid_{daily,monthly,annual}/MIROC6/ssp370/bcsd/tasmax/40.0_50.0_-125.0_-115.0/1981_2010_1981_2099.zarr' So, the changes being:

  • add downscale method between the ssp and the variable name
  • changing the time period string into the training period and future period separated by a _ instead of a /
  • Making the store be located in the time period directory instead of a subdirectory.

Do these look right @jhamman and @norlandrhagen? These changes percolate not just for the pyramids but for all of the zarr stores, which should be easy because everything is housed in the paths.py! Woohoo!

Also, I'm assuming these tweaks will be seamless with the other workflows, but we'll know soon!

@orianac
Copy link
Member Author

orianac commented Feb 3, 2022

Also I just had a thought that fixing (1) might help with memory issues (I feel like those were cropping up at some point in the obs step @norlandrhagen? Maybe that was in a different spot so disregard if I have it confused) as the current implementation doesn't have a full_time chunked version of obs to work with.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant