Target store returned from StoreToZarr
without waiting on completion of StoreDatasetFragments
#564
Labels
bug
Something isn't working
This is my mistake. When I merged #562, I did not understand that Python line numbers were not used for determining sequencing of the Beam DAG. I now see that by not chaining the returned
target_store
onto the output PCollection ofStoreDatasetFragments
herepangeo-forge-recipes/pangeo_forge_recipes/transforms.py
Lines 484 to 485 in 2739f22
Beam optimizes by returning
target_store
beforeStoreDatasetFragments
is complete. AFAICT the solution discussed in #556 (comment) makes the most sense. To summarize:target_store
fromStoreDatasetFragments
so that it emits a PCollection of lengthn
target_storesxref leap-stc/cmip6-leap-feedstock#9 (comment)
The text was updated successfully, but these errors were encountered: