Using a ZipStore instead of FSStore

[jni]

Hello!

I'm trying to write an ome-zarr for a small-ish dataset — ~90MB compressed. Because it's much easier to share single files than folders I want to try using a ZipStore. However, when I use a ZipStore zarr, napari-ome-zarr doesn't seem to be able to open the file. I'm not sure whether this is an issue with the plugin, with ome-zarr-py, or (really quite probable) my own confusion — I can't seem to even find the array in the ZipStore manually. 😬

Here's a small distillation of what I'm trying to do. In the code below I create two zarr files, one using ome_zarr.io.parse_url, and another using zarr.ZipStore. The code runs and writes the image to the ZipStore, but then I can't open it with napari-ome-zarr.

import shutil
from pathlib import Path
import numpy as np
import zipfile
from numcodecs import Zlib

from ome_zarr.writer import write_image
from ome_zarr.io import parse_url
import zarr

norm_path = Path('test-89043gpm0cu982.ome.zarr')
zip_path = Path('test-89043gpm0cu982.ome.zarr.zip')

shutil.rmtree(norm_path, ignore_errors=True)
shutil.rmtree(zip_path, ignore_errors=True)

try:
    arr = np.random.randint(0, 256, size=(3, 512, 512), dtype=np.uint8)
    scale = [0.2, 0.23, 0.23]

    coordtfs = [[{'type': 'scale', 'scale': scale}]]
    axes = [{'name': 'z', 'type': 'space', 'unit': 'millimeter'},
            {'name': 'y', 'type': 'space', 'unit': 'millimeter'},
            {'name': 'x', 'type': 'space', 'unit': 'millimeter'}]

    # Code just as in the docs
    store = parse_url(norm_path, mode='w').store
    root = zarr.group(store=store)
    write_image(arr, root, scaler=None, chunks=(1,) + arr.shape[1:], axes=axes,
                coordinate_transformations=coordtfs,
                storage_options={'compressor': Zlib(level=6)})
    root.attrs['omero'] = {
        'channels': [{
            'color': 'fd4762',
            'window': {'start': 45, 'end': 245},
            'label': 'image',
            'active': True,
        }],
    }


    # Now, same thing, but using ZipStore
    store_zip = zarr.ZipStore(zip_path, compression=zipfile.ZIP_DEFLATED, mode='w')
    root_zip = zarr.group(store=store_zip)
    write_image(arr, root_zip, scaler=None, chunks=(1,) + arr.shape[1:],
                axes=axes,
                coordinate_transformations=coordtfs,
                storage_options={'compressor': None})
    root_zip.attrs['omero'] = {
        'channels': [{
            'color': 'fd4762',
            'window': {'start': 45, 'end': 245},
            'label': 'image',
            'active': True,
        }],
    }

    import napari

    viewer = napari.Viewer()

    # works fine
    viewer.open(norm_path, plugin='napari-ome-zarr')
    # fails — plugin returns empty list
    viewer.open(zip_path, plugin='napari-ome-zarr')

    napari.run()

except:
    shutil.rmtree(norm_path, ignore_errors=True)
    shutil.rmtree(zip_path, ignore_errors=True)
    raise
finally:
    shutil.rmtree(norm_path, ignore_errors=True)
    shutil.rmtree(zip_path, ignore_errors=True)

Any advice would be appreciated! If there's a fix to be made somewhere in the code, as always, I'm happy to contribute, if someone can point me in the right direction! 🙏

[joshmoore]

jni commented yesterday
I'm not sure whether this is an issue with the plugin, with ome-zarr-py, or (really quite probable) my own confusion — I can't seem to even find the array in the ZipStore manually. 😬

...you left one out: with the spec(s)! The background is that Zarr v2 doesn't properly specify zip storage, i.e., it's a zarr-python-only feature. Ergo we didn't work to include it within ome-zarr-py. I don't imagine it would be too much work on the ome-zarr-py to "just make it work" but I wouldn't consider that standardized.

A conceivable route (still not standardized) that wouldn't require code changes would be to use fsspec-magic (also not standardized) a la:

zip::file://{filepath}"

See an example here: https://github.com/zarr-developers/outreachy_2022_testing_zipstore/blob/main/real%20%20world%20data/main.py#L53

[jni]

@joshmoore this is terrible 😂 Does v3 have zip in the standard or also no? 😬

I'm starting to think I should use a (😱) tiff... 😅

[joshmoore]

this is terrible 😂

Well that seems dramatic... 😄

Does v3 have zip in the standard or also no? 😬

Not yet, no. It will need a champion. In general, I think focus is currently on ZEP0002/sharding to solve the big many-files problem rather than the smaller set-of-files-isn't-as-usable problem.

I'm starting to think I should use a (😱) tiff... 😅

To some extent, this might just be the case, if what you are trying to do is avoid the use of a complexer format that you don't fully need.

[jni]

Yeah my point is, if I have a single, small, single-scale image, even just a folder with two metadata files and a subfolder with a single chunk is way more complex than one file. I think for ome-zarr to be the format for sharing images, this needs to be solved...

[joshmoore]

Definitely agreed. I think the question is where (& when) that sits on the road between where we are and where we want to be.

[imagesc-bot]

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/saving-volumetric-data-with-voxel-size-colormap-annotations/85537/1

[imagesc-bot]

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/saving-volumetric-data-with-voxel-size-colormap-annotations/85537/13