Automate saving in loop

Question

I want to automate the following code.

df.f1 <- c2d(fdat.s1, 1, 15000)
save(df.f1,file="../RESULTS/nl_final__df_1_1.rda")
df.f2 <- c2d(fdat.s1, 15000, 30000)
save(df.f2,file="../RESULTS/nl_final__df_1_2.rda")
df.f3 <- c2d(fdat.s1, 30000, 45000)
save(df.f3,file="../RESULTS/nl_final__df_1_3.rda")

I tried doing this. But I struggling with how to change df.f1 automatically in the loop.

vec <- c(1,15000,30000,45000)
for(i in 1:3){
  df.f1 <- c2d(fdat.s1, vec[i], vec[i+1])
  save(df.f1,file=paste("../RESULTS/nl_mod_df_1_",i,".rda"))
}

Is there are any efficient way other than loop?

Frankly, [loops and `apply` are roughly the same performance-wise](https://stackoverflow.com/questions/2275896/is-rs-apply-family-more-than-syntactic-sugar/2276001#2276001), so "efficient" is a very loose term here. — r2evans, Aug 29 '18 at 19:03

score 0 · Answer 1 · answered Aug 29 '18 at 16:55

Perhaps a combination of preinstantiating the datasets with a call to get is what you're after?

df.f1 <- c2d(fdat.s1, 1, 15000)
df.f2 <- c2d(fdat.s1, 15000, 30000)
df.f3 <- c2d(fdat.s1, 30000, 45000)

lapply(
  c(1, 2, 3),
  function(i){
    save(get(paste0("df.f", i)), file = paste("../RESULTS/nl_mod_df_1_", i, ".rda"))
  }
)

r2evans · Answer 2 · 2018-08-29T19:28:48.573

Edit: up front, efficiency is a relative term. Is there something wrong with your current method? This answer was written on the premise that you do things with multiple datasets (even if subsets of the same original set), but since then it seems that working with all data in a list is not tenable (due to data size). This last constraint negates the advantage of working with a list-of-frames, but I'll keep the answer here for the record.

I suggest dealing with lists (ref: How do I make a list of data frames?). Furthermore, it might be done with mapply.

First let's make a vector of the "breaks" (as you did):

brks <- seq(0L, 45000L, by=15000L)
brks
# [1]     0 15000 30000 45000

("How" you generate it is not critical, just that you have a vector that includes each of the break-points, including as well the first and last.) We'll be using "all but the last" and "all but the first" in two separate vectors, so you can see that we'll be spanning the first element of each, then the second element of each, etc:

brks[-length(brks)]
# [1]     0 15000 30000
brks[-1]
# [1] 15000 30000 45000

This isn't technically a requirement, though. My assumption is that your ranges should be contiguous, so using a vector to do with will guarantee continuity and safe-guard against updating one vector and not the other. You can certainly do (instead) two independent vectors.

mapply is a "zipper"-like function, where mapply(fun, 1:3, 11:13, SIMPLIFY=FALSE) is effectively list(fun(1,11), fun(2,12), fun(3,13)).

ret <- mapply(function(a,b) c2d(fdat.s1, a, b),
              brks[-length(brks)], brks[-1],
              SIMPLIFY=FALSE)

(SIMPLIFY=FALSE keeps it as a list. So many of the *apply functions like reducing things to vectors or matrices if given the chance, which can be problematic if that is not what you are expecting. To further complicate it, we see sapply(..., simplify=FALSE) and mapply(..., SIMPLIFY=FALSE), so slightly-different options for each.)

At this point, you can loop over the data and do whatever you need, including:

mapply(function(dat, nm) save(dat, file=nm),
       ret, sprintf("../RESULTS/nl_mod_df_1_%d.rda", brks[-length(brks)]))

Caveat emptor: untested, since I don't know what c2d is and I don't have your data. The concept is solid enough, I believe.

Thanks for the answer. If I understand correctly, you are saving everything in a list using mapply. If yes, then that is an issue. These are huge files (GB) and the list would huge for my computer to handle. Because of that, I am breaking 1:45000 into 3 parts. and saving them individually. — vikram, Aug 29 '18 at 17:57
Then the singular call to `c2d(...)` can be turned into a composite, `save(c2d(...),file="...")`. This suggests `mapply(function(a,b,nm) save(c2d(...),file=nm), brks[...], brks[...], sprintf(...))`. — r2evans, Aug 29 '18 at 18:50

Automate saving in loop

2 Answers2