CZI EOSS Cycle 6 - Discussion

[MSanKeys963]

Hi all! 👋🏻

CZI has announced the RFA for Essential Open Source for Science Cycle 6 (EOSS6) - the deadline for LOI is in 3 weeks, i.e. October 17 2023 (17:00 PT).

Check here: https://chanzuckerberg.com/rfa/essential-open-source-software-for-science/

I’m creating this issue to surface the interests and have a coordinated discussion for the application.

From the RFA - the acceptable use of funds are:

• Salary support for staff (full-time, part-time, or contract): developers, contributors, technical writers, community managers, product managers, project managers, user experience researchers, community educators, or other roles that directly support the software project(s);
• Hackathons, sprints, outreach, or other forms of community engagement and support for community participation;
• Operational needs such as cloud computing, storage, networking, or continuous integration services; and
• Support for work that bridges software projects or ecosystems, including better coordination across software projects that are similar, dependent on one another, or frequently used together.

I want to use this issue as a central point of discussion for all matters related to the application. If you/your project has some great ideas that we should consider, please let us know your thoughts.

CC: @zarr-developers/steering-council

[normanrz]

Thanks for starting this conversation. I am happy to contribute to the grant application.

My wishlist for the grant:

• Continued salary for community manager
• Funds for organizing ZarrCon (incl. travel grants)
• Spec and dev work
  • Zip stores
  • Consolidated metadata
  • Sparse arrays
  • Multiscales

[ivirshup]

Over at scverse we were thinking of doing a zarr-y EOSS proposal. What we want is one persons time to do:

• ZEP 3 (variable length chunks) python + javascript implementation
• ZEP 7 (strings) python + javascript implementation
• Building features in AnnData on top of these (better sparse support, kerchunk based concatenation, v3 support, maaaaybe index structures?)

I've previously discussed this with @MSanKeys963, where we were thinking of maybe doing a single "zarr for biomedical" grant with this + community things. But maybe we'd also do these as separate grants if the pool is too small.

I've heard from bioconductor people that the EOSS program likes complimentary projects, so I'd definitely want to point at other grant proposals and have a story about synergies.

I think @grimbough mentioned possibly wanting to do some rarr work?

[grimbough]

Thanks for the mention @ivirshup and bringing this thread to my attention.

Yes, I'm thinking off submitting an EOSS proposal around Zarr and Bioconductor. Initial thoughts are:

• Continuing development for Rarr and providing support for Zarr v3
• Providing an array-like interface in R i.e. x[1,4,5] semantics for operation on a Zarr-backed array object
• Delayed operation support via DelayedArray
• Support for domain specific datatypes built on Zarr e.g. OME-ZARR

I expect we would submit this as a single, stand-alone, proposal but would be very happy to collaborate and highlight potential links to others.

[melonora]

Thanks @giovp bringing this to my attention. How do all of you think of providing hybrid compression support?
The usecase is as follows: imaging of tissues usually has quite some margin around the tissues. While the actual tissue should be lossless compressed, chunks purely containing background could be completely compressed away. I think this would save quite some storage. @FrancescAlted I know I still need to send you some test data for this, just travelling a lot at the moment.

[nvictus]

zarr for biomedical

Sounds like a great idea!

Building features in AnnData on top of these (better sparse support, kerchunk based concatenation, v3 support, maaaaybe index structures?)

@manzt has worked on ZEP 1 (v3) and ZEP 2 (sharding) implementations and javascript stuff. An idea we've recently been tossing around is about implementing a Zarr-native sparse chunk codec. We also want to build full Zarr support into cooler (which is always sparse).

Maybe there are opportunities to synergize on these issues both upstream and across downstream projects.

[joshmoore]

T minus 5 days...

Obviously, a big thumbs up to all that was written above, and good luck to anyone who is applying. I haven't seen anything that needs my immediate attention, but if there is, my apologies; please let me know. (I'll be watching this space, but you can also contact me directly.)

I'll repeat here what I've said elsewhere:

• I fully support the idea of more projects being funded to work on/with Zarr.
• For grant-purposes, though, my assumption is that it's in your best interest to clearly identify an additional project that you are trying to bring into the EOSS fold. In my mind, this would include implementations that live outside of the org -- https://github.com/zarr-developers.
• In fact, based on recent NumFOCUS discussions, if you are explicitly writing a "Zarr" proposal (i.e. solely working on projects under the org), you should probably get in touch with the steering council ([email protected]) since there has been some recent confusion about applications from third-parties.
• That being said, I don't think additionally listing a project under the org is an issue whatsoever.
• Finally, from my side, I will only be applying for community management funds and no technical development if that helps clarify issues for anyone.

[ivirshup]

So scverse's proposal now covers the dev work I said above but added more work on building out analysis features on top of this. This will include sparse work, probably a chunked sparse implementation (likely some combination of: "Logical chunking of sparse arrays" and "Logical chunking of storage arrays").

[ivirshup]

@grimbough @nvictus, are you planning to go ahead with your applications?

[nvictus]

Our plan is to submit a LOI focusing on upgrades to cooler, including full zarr and xarray support and other improvements to stack-integration. It would be great to make this synergistic with scverse's more foundational proposal about sparse chunking (and some of the recent Sparse Array work with Scientific Python, which got CZI funding) and the overall goal of supporting "work that bridges software projects or ecosystems, including better coordination across software projects that are similar, dependent on one another, or frequently used together". Time is ticking, but happy to coordinate and help however I can.

[grimbough]

Yes, I've submitted an LOI focused on the topic I mentioned earlier.

[joshmoore]

🤞🏽

[ivirshup]

scverse is going for a full application. Should we try and do some further coordination here?

As a heads up, I'm traveling from the 12th to 30th so would be keen on coordinating sooner rather than later!

[joshmoore]

Hi @ivirshup. Same for the community management proposal. Happy to discuss ASAP incl. tomorrow. Unlikely to find a time for everyone tomorrow, but just in case: 12-2p and after 8p (Berlin) would work for me.

Pinging others who have spoken up here: @grimbough @nvictus @melonora @normanrz and additionally @tischi.

[nvictus]

Same here. Too late for today I guess, but tomorrow (Friday) is off for me so I can make time.

[joshmoore]

I'm online now working on the application if anyone is interested in jumping on a call. Ping on https://app.gitter.im/#/room/#zarr-developers_community:gitter.im

[grimbough]

How did people get on? My Rarr proposal was not funded. Hopefully some of the other Zarr related proposals were more successful.

[joshmoore]

Ah, thanks for the ping, @grimbough. The community manager / ZarrCon proposal was also not funded. There was a Fiji proposal that was accepted (🎉) I don't think I have heard from any others.