Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support external lz4, zstd, blosc, zlib #464

Open
haampie opened this issue Sep 25, 2023 · 5 comments
Open

Support external lz4, zstd, blosc, zlib #464

haampie opened this issue Sep 25, 2023 · 5 comments

Comments

@haampie
Copy link

haampie commented Sep 25, 2023

Problem description

This package seems to unconditionally build vendored copies of lz4, zstd and blosc. This makes it hard to

  1. integrate this package with other libraries that pull in any of these libraries at a different version -- if it were external, one version could be used for all;
  2. update or patch binary dependencies in case of CVEs;
  3. deal with alternative providers, e.g. zlib-ng instead of zlib.

Also, the "build system" seems to be effectively native only (cpuinfo based), and very much x86 centered. That also wouldn't be a big deal if I could point numcodes to pre-compiled external versions of lz4, zstd and blosc.

Can you please make this package work with an external/system lz4, zstd, and blosc? That'd be very helpful, thanks.

Probably this means using a better build system (maybe meson?) instead of a handwritten setup.py.

@haampie haampie changed the title Support external lz4, zstd, blosc Support external lz4, zstd, blosc, zlib Sep 25, 2023
@martindurant
Copy link
Member

I have mentioned elsehwere, but the package cramjam includes several useful and important compressors in one install, and ther tend to have good preformance compared to other versions. It does not support blosc, but that could be requested and would make numcodecs a much much simpler package to build and distribute.

@joshmoore
Copy link
Member

joshmoore commented Sep 25, 2023

Hi @haampie. I don't think anyone will disagree with your thesis here. It's more a matter of making it happen. Helping hands and suggestions like @martindurant's on getting us there are very welcome!

(e.g. see #274)

@BwL1289
Copy link

BwL1289 commented Aug 23, 2024

Linking #254. I know this is quite old. I'm experiencing this as well.

@milesgranger
Copy link

I learned about this issue thru one on cramjam (milesgranger/cramjam#110); I'm fixing to release 2.8.4 w/ blosc2 support; it's available now for testing w/ 2.8.4rc3 on PyPI. It already supports lz4, zstd and others in a small and easy install.

It would be interesting to know if this helps at all here.

@martindurant
Copy link
Member

Absolutely, thanks @milesgranger . The code here still references blosc1, so some compat work would be needed I think.

I really think that there is no need for numcodecs to be repeating the build process for these standard compressors and/or relying on system libraries. Some things will still need cython for custom algorithms, but the less the better.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants