R function scope and parallelism

Question

Consider the following function definitions

library(doParallel)
f_print <- function(x)
{
  print(x)
}
f_foreach <- function(l)
{
  foreach (i=l) %do%
  {
    f_print(i)
  }
}

f_foreach_parallel <- function(l)
{
  doParallel::registerDoParallel(1)
  foreach (i=l) %dopar%
  {
    f_print(i)
  }
}

Function use :

> f_foreach(c(1,2))
[1] 1
[1] 2
[[1]]
[1] 1

[[2]]
[1] 2

> f_foreach_parallel(c(1,2))
 Show Traceback

 Rerun with Debug
 Error in { : 
  task 1 failed - "impossible de trouver la fonction "f_print"" 
  [Error: could not find function f_print]
>

Can you help explain why the f_print() is not visible when parallelism is involved in foreach ? How can we use f_print() in this paralleled foreach ?Any documentations related to this point ?

The second function works for me without error. I'm running the latest of each package (doParallel 1.0.11 and foreach 1.4.3) on R 3.4.2. — lmo, Oct 06 '17 at 10:29
That's baffling. I was using doParallel_1.0.10 and foreach_1.4.3. I just updated to the same latest version as yours and the problem is still there. Any thoughts ? — Kenny, Oct 06 '17 at 10:53
That is strange. I just re-ran the code with a fresh version of R and did not get an error. Are you working in Windows? If so, there is some possibility that you'll have to export the function like this: `foreach (i=l, .export=c("f_print")) %dopar%`. If this works, then the issue has to do with the difference between `snow` and `mcapply` where the first works on all OS's and the second only works in *nix's. I am running openSuse linux. — lmo, Oct 06 '17 at 11:00
Possible duplicate of [could not find function inside foreach loop](https://stackoverflow.com/questions/4765256/could-not-find-function-inside-foreach-loop) — F. Privé, Oct 06 '17 at 11:03
You hit the spot. I use Windows and indeed _export_ solves it. That's some twist there. Also, `foreach` with parallel does not return list as normal `foreach` — Kenny, Oct 06 '17 at 12:28

score 0 · Answer 1 · answered Oct 23 '17 at 23:45

In addition to what has already been said in the comments of the top post, especially the one on specifying .export, when using the doFuture package your code will indeed work regardless of parallel backend, operating system, and .export. Here's an adapted version of your setup:

f_print <- function(x) {
  print(x)
}

f_foreach <- function(l) {
  foreach(i=l) %do% {
    f_print(i)
  }
}

f_foreach_dopar <- function(l) {
  foreach(i=l) %dopar% {
    f_print(i)
  }
}

Instead of doing:

library("doParallel")

## Setup PSOCK workers (just as on Windows)
workers <- parallel::makeCluster(1L, outfile = "")
registerDoParallel(workers)

f_foreach_dopar(c(1,2))
## Error in { : task 1 failed - "could not find function "f_print""

you can do:

library("doFuture")
registerDoFuture()

## As above
workers <- parallel::makeCluster(1L, outfile = "")
plan(cluster, workers = workers)

f_foreach_dopar(c(1,2))
## [1] 1
## [1] 2
## [[1]]
## [1] 1
## 
## [[2]]
## [1] 2

The reason why this works is that doFuture does a more thorough search to identify global variables (here f_print()).

PS. The reason for outfile = "" is so that stdout/stderr output (e.g. as from print()) is actually displayed. Redirecting stdout/stderr in parallel processing, which I don't recommend, is a whole different discussion, but I'll assume you used print() just for your example.

R function scope and parallelism

1 Answers1

Linked