Add .nwb to known Zarr suffixes; add tests for NWB Zarr backend #1312

CodyCBakerPhD · 2023-07-18T19:39:56Z

This seems to have worked fine for me on a local upload attempt to staging (validation had to be suppressed until we get NeurodataWithoutBorders/nwbinspector#388 and the integration follow-up through)

At least getting the seeds of this PR started

codecov · 2023-07-18T19:59:42Z

Codecov Report

Attention: 1 lines in your changes are missing coverage. Please review.

Comparison is base (72816ab) 88.51% compared to head (a7d16aa) 67.93%.

Files	Patch %	Lines
dandi/tests/fixtures.py	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #1312       +/-   ##
===========================================
- Coverage   88.51%   67.93%   -20.58%     
===========================================
  Files          77       77               
  Lines       10492    10480       -12     
===========================================
- Hits         9287     7120     -2167     
- Misses       1205     3360     +2155

Flag	Coverage Δ
unittests	`67.93% <80.00%> (-20.58%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

CodyCBakerPhD · 2023-07-18T20:06:07Z

The current CI error

TypeError: Can't instantiate abstract class NWBZarrIO with abstract method can_read

has a known cause and will likely need to await the upcoming release of hdmf_zarr (this week sometime)

dandi/consts.py

CodyCBakerPhD · 2023-08-11T15:27:49Z

@yarikoptic Trying the main branch (not this one) with .nwb.zarr gave the following error during dandi upload -i dandi-staging to destination https://gui-staging.dandiarchive.org/dandiset/204919/draft/files?location=test_read_nwbfile

501 Server Error: Not Implemented for url: https://dandi-api-staging-dandisets.s3.amazonaws.com/zarr/d...

Worth mentioning it began upload and proceeded find until 99% done when the error happened

log: 20230811152021Z-372.log

yarikoptic · 2023-08-11T16:41:12Z

eh -- I spent some time trying to see smth in the logs

your activity came at unfortunate time when I had to relogin the client so I do not have logs for

-rw------- 1 dandi dandi-internal 112444 Aug 11 11:00 20230811-1001.log
-rw------- 1 dandi dandi-internal      0 Aug 11 11:01 20230811-1101.log
-rw------- 1 dandi dandi-internal      0 Aug 11 12:01 20230811-1201.log
-rw------- 1 dandi dandi-internal      0 Aug 11 12:20 20230811-1220.log
-rw------- 1 dandi dandi-internal  45857 Aug 11 12:37 20230811-1224.log

today

in papertrail -- nothing informative if searched by dd868c2b-0a3b-44cb-afdb-eb62f07a701b , nothing for "implemented" or "exception" (only older ones). @dandi/archive-admin might have better skills/tools to see if anything in the logs.

or if you retry again -- I might spot smth in the fresh dumps for heroku which I should be getting now.

yarikoptic · 2023-08-11T16:50:26Z

The current CI error
TypeError: Can't instantiate abstract class NWBZarrIO with abstract method can_read
has a known cause and will likely need to await the upcoming release of hdmf_zarr (this week sometime)

FWIW, in dev-deps CI run we see the same Error and we do

        pip install git+https://github.com/jaraco/keyring
        pip install git+https://github.com/NeurodataWithoutBorders/nwbinspector
        pip install git+https://github.com/NeurodataWithoutBorders/pynwb

so we do not install development hdmf as well. Do you think we should?

Inspired by #1312 and ongoing work for Zarr based backend for NWB.

CodyCBakerPhD · 2023-08-11T16:52:18Z

I just tried again (from the save environment) and did not get the previous error

Asset is now visible: https://gui-staging.dandiarchive.org/dandiset/204919/draft/files?location=test_read_nwbfile

CodyCBakerPhD · 2023-08-11T16:52:54Z

BTW the 'environment' I'm using right now is just the default/base on the DANDI Hub

yarikoptic · 2023-08-11T17:09:07Z

so we do not install development hdmf as well. Do you think we should?

trying in #1320 and it leads to an error (posted there)

I just tried again (from the save environment) and did not get the previous error

well, would need 1 more fresh .nwb.zarr I guess ;-)

Inspired by #1312 and ongoing work for Zarr based backend for NWB.

dandi/consts.py

yarikoptic

Let's make it more flexible and up to date

yarikoptic · 2024-12-02T15:15:20Z

dandi/tests/fixtures.py

@@ -234,6 +234,17 @@ def organized_nwb_dir2(
        ),
        **simple1_nwb_metadata,
    )
+    make_nwb_file(
+        tmp_path / "simple1_zarr.nwb",


I think we ended up with .nwb.zarr, right @rly ?

yarikoptic · 2024-12-02T15:16:00Z

dandi/pynwb_utils.py

+    filename: StrPath,
+    *args: Any,
+    cache_spec: bool = False,
+    backend: Literal["hdf5", "zarr"] = "hdf5",


Let's make it more explicit

Suggested change

backend: Literal["hdf5", "zarr"] = "hdf5",

backend: Literal["auto", "hdf5", "zarr"] = "auto",

with "auto" choosing based on extension.

CodyCBakerPhD added 5 commits July 18, 2023 15:11

add .nwb to known zarr suffixes

b31e677

add zarr backend to pynwb_utils

5049999

add extra zarr file to existing directory fixture

9bfb9ca

add hdmf_zarr to requirements

5d48f40

black and import

71e8c3d

yarikoptic reviewed Aug 7, 2023

View reviewed changes

dandi/consts.py Outdated Show resolved Hide resolved

yarikoptic added a commit that referenced this pull request Aug 11, 2023

Install also hdmf from git in the dev-deps CI run

35b37e0

Inspired by #1312 and ongoing work for Zarr based backend for NWB.

yarikoptic mentioned this pull request Aug 11, 2023

Install also hdmf from git in the dev-deps CI run #1320

Merged

yarikoptic added a commit that referenced this pull request Nov 1, 2023

Install also hdmf from git in the dev-deps CI run

ed2803c

Inspired by #1312 and ongoing work for Zarr based backend for NWB.

CodyCBakerPhD commented Jan 4, 2024

View reviewed changes

dandi/consts.py Outdated Show resolved Hide resolved

CodyCBakerPhD added 2 commits January 4, 2024 11:33

Update dandi/consts.py

14c9c1e

Merge branch 'master' into try_nwb_zarr

a7d16aa

yarikoptic requested changes Dec 2, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add .nwb to known Zarr suffixes; add tests for NWB Zarr backend #1312

Add .nwb to known Zarr suffixes; add tests for NWB Zarr backend #1312

CodyCBakerPhD commented Jul 18, 2023 •

edited

Loading

codecov bot commented Jul 18, 2023 •

edited

Loading

CodyCBakerPhD commented Jul 18, 2023

CodyCBakerPhD commented Aug 11, 2023 •

edited

Loading

yarikoptic commented Aug 11, 2023

yarikoptic commented Aug 11, 2023

CodyCBakerPhD commented Aug 11, 2023 •

edited

Loading

CodyCBakerPhD commented Aug 11, 2023

yarikoptic commented Aug 11, 2023

yarikoptic left a comment

yarikoptic Dec 2, 2024

yarikoptic Dec 2, 2024

	backend: Literal["hdf5", "zarr"] = "hdf5",
	backend: Literal["auto", "hdf5", "zarr"] = "auto",

Add .nwb to known Zarr suffixes; add tests for NWB Zarr backend #1312

Are you sure you want to change the base?

Add .nwb to known Zarr suffixes; add tests for NWB Zarr backend #1312

Conversation

CodyCBakerPhD commented Jul 18, 2023 • edited Loading

codecov bot commented Jul 18, 2023 • edited Loading

Codecov Report

CodyCBakerPhD commented Jul 18, 2023

CodyCBakerPhD commented Aug 11, 2023 • edited Loading

yarikoptic commented Aug 11, 2023

yarikoptic commented Aug 11, 2023

CodyCBakerPhD commented Aug 11, 2023 • edited Loading

CodyCBakerPhD commented Aug 11, 2023

yarikoptic commented Aug 11, 2023

yarikoptic left a comment

Choose a reason for hiding this comment

yarikoptic Dec 2, 2024

Choose a reason for hiding this comment

yarikoptic Dec 2, 2024

Choose a reason for hiding this comment

CodyCBakerPhD commented Jul 18, 2023 •

edited

Loading

codecov bot commented Jul 18, 2023 •

edited

Loading

CodyCBakerPhD commented Aug 11, 2023 •

edited

Loading

CodyCBakerPhD commented Aug 11, 2023 •

edited

Loading