Simulations with map family

[jpiippon]

Edited: I have three questions and I'd be grateful if you can help me with them.

1. I have a disk.frame object containing around 2m rows and I want to do simulations for each of the rows and variables. I demonstrate this with disk.frame object dat.df that includes columns var_a, var_b and var_c. However, collecting does not work if I include the second last row that contains the function pmap(). Also, when I try to run all the other rows for my original data, my 32GB RAM will be full

library(dplyr)
library(purrr)
library(disk.frame)
library(truncnorm)

dat.df <- tibble(
  var_a = c(1:5),
  var_b = c(101:105),
  var_c = c(11:15)
)

setup_disk.frame()
dat.df1 = as.disk.frame(dat.df)

dat.df2 <- dat.df1 %>% 
  mutate(mean_val = rep(1, nrow(.))) %>%   # 1)
  mutate(var_rnorm = map(.x = mean_val, ~ rnorm(1000, mean = .x, sd = 0.07))) %>%  # 2)
  mutate(sim_var_a = map2(.x = var_a, .y = var_rnorm,  ~ (.x * .y))) %>%  # 3)
  mutate(across(var_b:var_c, ~  (0.1 + 0.013 * .x))) %>%  # # 4) preseve the names var_b and var_c
  mutate(var_multiplication = map2(.x =sim_var_a, .y =  var_b, ~ (.x * .y)) ) %>%  # 5)
  mutate(med_result = map(.x = var_multiplication, ~ median(.x))) %>% # 6)
  #mutate(this_does_not_work = pmap(., ~ rtruncnorm(1000, a = var_a, b=var_b, mean =var_c))) %>% # 7)
  dplyr::select(med_result) %>% # 8)
  collect

where

1. just column containing 1 1 1 1
2. normal distribution, n = 1000, mean = mean_val we created above, standard deviation = 0.07
3. multiply var_a with var_rnorm
4. multiply variables var_b =(0.1 + 0.013) * var_b and var_c = (0.1 + 0.013) * var_c
5. multiply simulated variable a (contains 1000 obs for each row) with var_b that has been multiplied with (0.1 + 0.013) above
6. take a median. Now we would have one value for each row
7. trying to create a truncated normal distribution for each row with pmap but no luck
8. select only the variable needed to avoid RAM problems

So, is it possible to use pmap or how do # 7 with disk.frame or use more than 2 columns in any computation? What might be the reason for these errors I get when I try to run a very similar example like this (without the pmap row) with my original data?

Using as.character() on a quosure is deprecated as of rlang 0.3.0.
Please use as_label() or as_name() instead.

or

This warning is displayed once per session.Error in value[3L] : disk.frame has detected a syntax error in ~

2. I have created .fst files earlier with as.disk.frame() or with csv_to_disk.frame but what is the best way to access them when I start coding next time? Now I have to do disk.frame conversion every time but I am sure there is a better way.

[xiaodaigh]

3. Is fairly easy to answer. Just provide a path to where you want to store your disk.frame, then you can load it back

adiskf = as.disk.frame(iris, outpath = "somewhere")

adiskf_reloaded = disk.frame("somewhere")

For 2, I don't really get what u mean. Perhaps you can create a Minimial workable example with code that can actually be run? The code needs to also generate the data so I don't have to fiddle.

For 1 and also as an example of how to create a minimal working example (MWE), see how the code I made can run and the data is generate within the code so the people trying to answer your question doesn't have to do too much?

Seems like you just need to add collect. The code works. But I suspect the use case you have not result in much speed up anyway.

library(dplyr)
library(purrr)

library(disk.frame)
setup_disk.frame()

dat.df1 = as.disk.frame(dat.df)

dat.df2 <- dat.df1 %>% 
  mutate(mean_val = rep(1, nrow(.))) %>%   # 1)
  mutate(var_rnorm = map(.x = mean_val, ~ rnorm(1000, mean = .x, sd = 0.07))) %>%  # 2)
  mutate(sim_var_a = map2(.x = var_a, .y = var_rnorm,  ~ (.x * .y))) %>%  # 3)
  mutate(across(var_b:var_c, ~  (0.1 + 0.013 * .x))) %>%  # # 4) preseve the names var_b and var_c
  mutate(var_multiplication = map2(.x =sim_var_a, .y =  var_b, ~ (.x * .y)) ) %>%  # 5)
  mutate(med_result = map(.x = var_multiplication, ~ median(.x))) %>% 
  collect

[jpiippon]

Thanks, I tried to update my original question. Does it make more sense now?