how to get the max value of a column

[SMousavi90]

Considering this line of code which in dataframes returns the max value of the "date" column, how can we do the same with diskframe? I tried multiple ways but none of them works.

max_date <- max(my.df$date,na.rm=TRUE)

this is the one that didn't work:

max <- (my.df %>% summarise(max(date, na.rm=T)))[[1]]

it returns an integer which is not a date obviously.

[xiaodaigh]

you might be just missing collect which is needed.

library(disk.frame)

i = as.disk.frame(iris)

i %>% 
  summarise(max(Sepal.Width, na.rm=T)) %>% 
  collect() %>% 
  pull()

[SMousavi90]

Thank you for your reply, but the this diskframe is a big diskframe with over 10 million rows. if I collect it, it takes minutes to return the value, is there any other alternatives that is faster?

[xiaodaigh]

Are you just collecting one value which is the result of the summarize? That's the best way. You can use the srckeep function to load only the column you want to use into memory which should be fast

i %>% 
  srckeep("Sepal.Width") %>%
  summarise(max(Sepal.Width, na.rm=T)) %>% 
  collect() %>% 
  pull()

[SMousavi90]

Yes I only need the max value of the date column. one cell only. still takes many time to do the operation even by using srckeep.

[xiaodaigh]

On my 6-core i7, this is as fast as it can be

   user  system elapsed 
   0.67    0.00    1.95 

So I don't have any more tricks up my sleeves

See MWE

library(disk.frame)
setup_disk.frame()

rdate <- function(x,
                  min = paste0(format(Sys.Date(), '%Y'), '-01-01'),
                  max = paste0(format(Sys.Date(), '%Y'), '-12-31'),
                  sort = TRUE) {
  
  dates <- sample(seq(as.Date(min), as.Date(max), by = "day"), x, replace = TRUE)
  if (sort == TRUE) {
    sort(dates)
  } else {
    dates
  }
  
}


my.df = as.disk.frame(
  data.frame(
    date = rdate(10e7),
    rand = runif(10e7)
  )
)

system.time(a <- my.df %>% 
  srckeep("date") %>% 
  summarise(max(date, na.rm=T)) %>% 
  collect() %>% 
  pull())