arrange all data

[arnaud-feldmann]

Hi,

Is an implementation of global arrange still intended for your package disk.frame ?
I understand that’s one of the complicated things to think about, due to the preference for not loading the data too much times.
Though, that's one of the things I'd need to have if I put my data that way

Thanks for your work.

[xiaodaigh]

Do you mean hard_arrange? It's implemented by a contributor Jacky Poon

library(disk.frame)
setup_disk.frame() # not needed for small datasets but just for show

a = as.disk.frame(iris)

# this will arrange (i.e. sort) the disk.frame on disk by Species
# WARNING: extremely slow for large datasets
a = disk.frame::hard_arrange(a, Species)

a_collected = collect(a, parallel = FALSE)

is.unsorted(a_collected$Species) # FALSE; so it's sorted

[arnaud-feldmann]

Thanks. I had seen the function but misunderstood it as a grouping function more than a sorting one.
I don't understand the doc through :

A hard_arrange is a sort by that also reorganizes the chunks to ensure that every unique grouping of 'by“ is in the same chunk. Or in other words, every row that share the same 'by' value will end up in the same chunk.

1. There is no "by" argument in the function, and the doc is a bit unclear. Do you mean the dots ?
2. The doc made me think hard_arrange is something like folders hierarchy for parquet files, that is, different folders that stand for variables. But then, what happens if i sort my chunks with a variables containing a very few possible values that would induce very big equivalence classes ? Will the chunks be possibly too big to be loaded or is there some kind of safety net ?

Thanks

[xiaodaigh]

Yeah. I see. the dots are the by. The syntax is should be the same as arrange.

2. Yes, it might get skewed. And no it's not like folder partition it's more like BigQuery's cluster. Some systems call it sharding. BTW {disk.frame} doesn't have folder partition yet.