The best way to distribute large scientific datasets is via the Cloud, in Cloud-Optimized formats 1. But often this data is stuck in legacy pre-Cloud file formats such as netCDF.
VirtualiZarr2 makes it easy to create "Virtual" Zarr stores, allowing performant access to legacy data as if it were in the Cloud-Optimized Zarr format, without duplicating any data.
Please see the documentation.
- Create virtual references pointing to bytes inside a legacy file with
open_virtual_dataset, - Supports a range of legacy file formats, including netCDF4 and HDF5,
- Combine data from multiple files into one larger store using xarray's combining functions, such as
xarray.concat, - Commit the virtual references to storage either using the Kerchunk references specification or the Icechunk transactional storage engine.
- Users access the virtual dataset using
xarray.open_dataset.
VirtualiZarr grew out of discussions on the Kerchunk repository, and is an attempt to provide the game-changing power of kerchunk but in a zarr-native way, and with a familiar array-like API.
You now have a choice between using VirtualiZarr and Kerchunk: VirtualiZarr provides almost all the same features as Kerchunk.
VirtualiZarr version 1 (mostly) achieves feature parity with kerchunk's logic for combining datasets, providing an easier way to manipulate kerchunk references in memory and generate kerchunk reference files on disk.
Future VirtualiZarr development will focus on generalizing and upstreaming useful concepts into the Zarr specification, the Zarr-Python library, Xarray, and possibly some new packages.
We have a lot of ideas, including:
- Zarr v3 support
- Zarr-native on-disk chunk manifest format
- "Virtual concatenation" of separate Zarr arrays
- ManifestArrays as an intermediate layer in-memory in Zarr-Python
- Separating CF-related Codecs from xarray
- Generating references without kerchunk
If you see other opportunities then we would love to hear your ideas!
- 2024/11/21 - MET Office Architecture Guild - Tom Nicholas - Slides
- 2024/11/13 - Cloud-Native Geospatial conference - Raphael Hagen - Slides
- 2024/07/24 - ESIP Meeting - Sean Harkins - Event / Recording
- 2024/05/15 - Pangeo showcase - Tom Nicholas - Event / Recording / Slides
This package was originally developed by Tom Nicholas whilst working at [C]Worthy, who deserve credit for allowing him to prioritise a generalizable open-source solution to the dataset virtualization problem. VirtualiZarr is now a community-owned multi-stakeholder project.
Apache 2.0
Footnotes
-
Cloud-Native Repositories for Big Scientific Data, Abernathey et. al., Computing in Science & Engineering. ↩
-
(Pronounced like "virtualizer" but more piratey 🦜) ↩