Cannot write DataFrames with more than 2^31 rows

[Para7etamol]

Sadly, JDF.jl depends on Blosc v1, which only supports 32bit adressing.

So this is impossible:

JDF.save(joinpath("/tmp", "test.jdf"), DataFrame(bytes=rand(Int8, 3_000_000_000)))

Please allow to disable the compression or to choose other algorithms. Or a way to split the JDF-files into partitions with < 2^31 rows.

Greetings
Para

[PallHaraldsson]

There's already https://juliahub.com/ui/Packages/General/Blosc2

Bloch2 is a new compression format, and software (also retaining old API), and it's 64-bit. The older is yes 32-bit, I think though should be able to go up to 4 GB, without changing the file format (but would need a fix in Blosh.jl, there's an issue on the limitation there).

[xiaodaigh]

working on this now. It's write it now but not read it yet. So hopefully I will find some time to get the reading working