2

I am just getting started with foreach & parallel and encountered some challenges with combining my outputs from each iteration.

Currently, in each iteration, I am outputting a set of variables that I would like to combine into a vector, so that I can obtain some summary stats for across all iterations. An example: 

foreach (s = 1:num) %dopar%{
var1 = ...
var2 = ...
var3 = ...}

Ultimately I'd like 3 vectors containing each iteration's produced output, so: var1 = c(var1 from s=1, var1 from s=2, var1 from s=3, ...)

I looked into the .combine option and attempted to create a function that would append, but found no success.

pogibas
  • 27,303
  • 19
  • 84
  • 117
user7044439
  • 25
  • 2
  • 5
  • What did your attempt look like? How did it fail? It's easier to help with a [reproducible example](https://stackoverflow.com/questions/5963269/how-to-make-a-great-r-reproducible-example). If you need to return more than one value per loop, you'll need to do so in a `list()` – MrFlick Sep 21 '17 at 21:31
  • 2
    Possible duplicate: https://stackoverflow.com/questions/19791609/saving-multiple-outputs-of-foreach-dopar-loop – MrFlick Sep 21 '17 at 21:31
  • 1
    Possible duplicate of [Saving multiple outputs of foreach dopar loop](https://stackoverflow.com/questions/19791609/saving-multiple-outputs-of-foreach-dopar-loop) – F. Privé Sep 22 '17 at 06:03

2 Answers2

6

The {} after foreach...%dopar% behaves like a function, that is, it will return a value whether you explicitly specify it or not. It is generally not specified but by specifying you can return whatever value you like

foreach(I=1:3) %dopar% { <complicated stuff> 
                         return(1) }

# [[1]]
# [1] 1

# [[2]]
# [1] 1

# [[3]]
# [1] 1

In your case, you want to return 3 vectors, unfortunately this is not possible

library(foreach)
library(doParallel)
cl <- makeCluster(detectCores())
registerDoParallel(cl)

foreach(I=1:8) %dopar% { v1 <- "ant"
                 v2 <- "pant"
                 return(v1, v2) }

Which results in the following error

Error in { : task 1 failed - "multi-argument returns are not permitted"

But you can make a list of your vectors, and return the value

foreach(I=1:8, .combine=rbind) %dopar% {  v1 <- "ant"
                                          v2 <- "pant"
                                          return(list(v1, v2)) }

         # [,1]  [,2]  
# result.1 "ant" "pant"
# result.2 "ant" "pant"
# result.3 "ant" "pant"
# result.4 "ant" "pant"
# result.5 "ant" "pant"
# result.6 "ant" "pant"
# result.7 "ant" "pant"
# result.8 "ant" "pant"

stopCluster(cl)
CPak
  • 13,260
  • 3
  • 30
  • 48
  • Thanks for this! I would prefer a solution that does not rely on another package and this answer would fit the bill. – user7044439 Sep 22 '17 at 20:22
  • Follow up on this solution - how would I access the resulting list? From the stacked lists, I would want to essentially recreate v1 by subsetting to the first column of the stacked list. – user7044439 Sep 22 '17 at 20:45
  • 1
    EDIT: I was able to subset by assigning the whole foreach to a variable. THanks for your help! – user7044439 Sep 22 '17 at 21:12
5

Solution using future:
doFuture is future's wrapper for foreach which I prefer instead of parallel.
Here we create data.frame res and you can access vector using res$vector1

library(doFuture)
registerDoFuture()
plan(multiprocess)

Nlength <- 1000
Nvector <- 3

res <- foreach(i = 1:Nvector, .combine = cbind) %dopar% {
    1:Nlength / pi * rnorm(1)
}
res <- data.frame(res)
colnames(res) <- paste0("vector", 1:Nvector)

dim(res)
[1] 1000    3
pogibas
  • 27,303
  • 19
  • 84
  • 117