You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We have the same target file (make_coarse_obs_path) for both get_coarse_obs_task and return_coarse_obs_full_time_task in bcsd_flow.py. I don't think this is right, unless you're intentionally trying to overwrite? But wouldn't prefect instead just always think that the task is cached because the target already exists by the time it gets to that task? Perhaps there is some special magic though that I don't see? (Update: adding the grid_spec and chunking_method back into the obs_identifier should help with this)
Right now the pyramid files are stored with a path schema that looks like this:
'az://flow-outputs/results_testing/pyramid_{daily,monthly,annual}/MIROC6/ssp370/tasmax/40.0_50.0_-125.0_-115.0/1981_2010/1981_2099/.pyr'.
We settled on changing that to: 'az://flow-outputs/results_testing/pyramid_{daily,monthly,annual}/MIROC6/ssp370/bcsd/tasmax/40.0_50.0_-125.0_-115.0/1981_2010_1981_2099.zarr' So, the changes being:
add downscale method between the ssp and the variable name
changing the time period string into the training period and future period separated by a _ instead of a /
Making the store be located in the time period directory instead of a subdirectory.
Do these look right @jhamman and @norlandrhagen? These changes percolate not just for the pyramids but for all of the zarr stores, which should be easy because everything is housed in the paths.py! Woohoo!
Also, I'm assuming these tweaks will be seamless with the other workflows, but we'll know soon!
The text was updated successfully, but these errors were encountered:
Also I just had a thought that fixing (1) might help with memory issues (I feel like those were cropping up at some point in the obs step @norlandrhagen? Maybe that was in a different spot so disregard if I have it confused) as the current implementation doesn't have a full_time chunked version of obs to work with.
We have the same target file (make_coarse_obs_path) for both get_coarse_obs_task and return_coarse_obs_full_time_task in bcsd_flow.py. I don't think this is right, unless you're intentionally trying to overwrite? But wouldn't prefect instead just always think that the task is cached because the target already exists by the time it gets to that task? Perhaps there is some special magic though that I don't see? (Update: adding the
grid_spec
andchunking_method
back into theobs_identifier
should help with this)Right now the pyramid files are stored with a path schema that looks like this:
'az://flow-outputs/results_testing/pyramid_{daily,monthly,annual}/MIROC6/ssp370/tasmax/40.0_50.0_-125.0_-115.0/1981_2010/1981_2099/.pyr'.
We settled on changing that to: 'az://flow-outputs/results_testing/pyramid_{daily,monthly,annual}/MIROC6/ssp370/bcsd/tasmax/40.0_50.0_-125.0_-115.0/1981_2010_1981_2099.zarr' So, the changes being:
_
instead of a/
Do these look right @jhamman and @norlandrhagen? These changes percolate not just for the pyramids but for all of the zarr stores, which should be easy because everything is housed in the
paths.py
! Woohoo!Also, I'm assuming these tweaks will be seamless with the other workflows, but we'll know soon!
The text was updated successfully, but these errors were encountered: