Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build rechunking tutorial #277

Closed
amsnyder opened this issue Mar 31, 2023 · 3 comments
Closed

Build rechunking tutorial #277

amsnyder opened this issue Mar 31, 2023 · 3 comments
Assignees

Comments

@amsnyder
Copy link
Contributor

amsnyder commented Mar 31, 2023

@rsignell-usgs is building a rechunking tutorial with the following structure:
(need to populate)

@gzt5142 is making sure the content populates a JupyterBook nicely. Current draft is here: https://gzt5142.github.io/DaskDataChunking/

Once the tutorial/book is complete, we will decide how to link to or incorporate it into the HyTEST JupyterBook.

@amsnyder
Copy link
Contributor Author

amsnyder commented Apr 10, 2023

@rsignell-usgs will be building the tutorial on NOAA GEFS retrospective data. The first two steps of his workflow (create individual file jsons and create consolidated metadata json/parquet file) will be redundant to what PUMP (@ted80810, @wdwatkins) has already done, so Rich won't need to actually run this code - just run it on a sample of files to test and make sure it works.

Rich will actually build out the steps of the workflow to rechunk the data from the consolidated metadata file because PUMP has not worked on this part yet. We can provide the rechunked dataset to PUMP as a value-added substitution when it is ready.

@amsnyder
Copy link
Contributor Author

Tasks:

@amsnyder
Copy link
Contributor Author

closing - replaced by #534

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants