Create an agenda for the tutorial

[negin513]

Since there are different topics/aspects that can be covered for the Dask tutorial and we have a limited time we need to define the scope of the tutorial and create a clear agenda for it.

From our conversation with @dcherian and @vanderwb : This going to be a half-day event with topics and sessions for the following audiences:
(a) folks who are Python-aware but essentially total novices to Dask.
(b) those who use Dask regularly but would like optimization guidance and tips/tricks.

The followings are my initial thoughts on what would be appropriate for this meeting:

• Introductory Dask + Xarray:

  • What is Dask
  • Dask +Xarray
  • dask-backed Xarray objects (lazy computations, actual values)
  • Distributed clusters
  • Extract Dask arrays from Xarray objects and use Dask array directly.
  • Dask Delayed to parallelize any code ( I think this will be important, but do we have time to include this?)

• Intermediate Topics:

  • Dask chunking schemes (performance and rechunking)
  • Apply unvectorized functions (apply_unfunc)
  • More advanced collection of custom operations (map_blocks, map_partitions, map_overlap do we have enough time for this?)
  • Blockwise computation

Since this is going to be a half-day tutorial, we need to be cautious of the time.

We are going to solicit more feedback on this from the community and ESDS forum.

[dcherian]

copying over @katiedagon's Zulip comment:

dask-jobqueue usage and best practices
monitoring and managing resource requests (e.g., workers, memory)
leveraging the dask dashboard
reading and writing data with xarray and dask, including chunking best practices
managing computation (e.g., when to .compute or .persist)

[negin513]

My proposed Agenda to be discussed with others:

1. Introductions to Dask

   1. What is Dask?
   2. Dask Collections
      • Introduction of Base Data types ( High Level Collections):
        • Dask DataFrames
        • Dask Arrays
        • Dask Bags.
      • Low-level collections (let's not touch for now) dask delayed stuff. * (possibly no time for this)
   3. Dask and Dask Cluster
      • Different types of clusters (https://docs.dask.org/en/stable/deploying.html)
      • managing computation (.persist or .compute)
      • Dashboard

2. Introductions on Xarray + Dask

   1. read/write netcdf files with Dask
   2. Dask with Xarray
      • Dask backed Xarray objects and operations
      • Extract Dask arrays from Xarray objects and use Dask array directly.
      • Dask Delayed to parallelize any code ( I think this will be important, but do we have time to include this?)
   3. Apply unvectorized functions (apply_unfunc) * (possibly no time for this)
   4. More advanced collection of custom operations (map_blocks, map_partitions, map_overlap do we have enough time for this?)

3. Best Practices for HPC environment:

   1. Monitoring Resources
      • dask-jobqueue usage and best practices
      • monitoring and managing resource requests (e.g., workers, memory)
      • dask dashboard * (possibly no time for this)
   2. Compute Allocation Usage
   3. Chunking best practices (Dask Chunking Schemes)
   4. Dask + GPUs (Time)

4. Misc. Stuff:

   1. How to parallelize a script with Dask? (From start to end?) One full example to end all the tutorial.