could not find function inside foreach loop

Question

I'm trying to use foreach to do multicore computing in R.

A <-function(....) {
    foreach(i=1:10) %dopar% {
    B()
    }
}

then I call function A in the console. The problem is I'm calling a function Posdef inside B that is defined in another script file which I source. I had to put Posdef in the list of export argument of foreach: .export=c("Posdef"). However I get the following error:

Error in { : task 3 failed - "could not find function "Posdef""

Why cant R find this defined function?

I would recommend searching for information regarding dynamic scoping with R. It's not at all intuitive and you'll find a lot of questions on SO that detail people having problems while defining a function within a function. — Brandon Bertelsen, Jan 23 '11 at 09:21
We would appreciate a reproducible example. Also, have you tried snowfall for multicoring? I find it more intuitive and easier to adapts (modifying apply functions). — Roman Luštrik, Jan 23 '11 at 11:50
same issue, not found any simple answer for now. I managed to make it work by calling on 4 cores foreach (j=1:N, .combine=rbind, .packages=c(VECTOR OF PACKAGES REQUIRED TO CALL ALL YOUR FUNCTIONS)) %dopar% fun(j). It improves the speed, but doesn't divide by 4 (by 2 only) I am not sure what I do so do not replicate this.. — RockScience, Mar 10 '11 at 12:37
I solved my problem by exporting every problematic functions with `clusterExport(cl, 'myMean')` suggested here https://stackoverflow.com/questions/17879766/variable-scope-in-boot-in-a-multiclustered-parallel-approach?noredirect=1&lq=1 — user3226167, Jan 09 '20 at 08:49

Paul Eastlund · Answer 1 · 2014-02-25T03:29:23.817

16

So I can reproduce this, for the curious:

require(doSNOW)
registerDoSNOW(makeCluster(5, type="SOCK"))
getDoParWorkers()
getDoParName()
getDoParVersion()

fib <- function(n) {
  if (n <= 1) { return(1) }
  return(fib(n-1) + fib(n-2))
}

my.matrix <- matrix(runif(2500, 10, 50), nrow=50)

calcLotsaFibs <- function() {
  result <- foreach(row.num=1:nrow(my.matrix), .export=c("fib", "my.matrix")) %dopar% {
    return(Vectorize(fib)(my.matrix[row.num,]))
  }
  return(result)
}

lotsa.fibs <- calcLotsaFibs()

I have been able to get around this by putting the function in another file and loading that file in the body of the foreach. You could also obviously move the function definition into the body of the foreach itself.

[EDIT -- I had previously suggested that perhaps .export doesn't work properly with function names, but was corrected below.]

edited Feb 25 '14 at 03:29

answered Apr 11 '12 at 21:21

Paul Eastlund

6,713
4
18
20

2

".export" can take function names. This is an interesting example, however, since I believe the problem is that the error occurs when `fib` makes a recursive call to itself, since `fib` is not in its own scope, as I explain in my answer. – Steve Weston Jul 26 '13 at 16:44
you mean that it could work if we wrap the custom function in a R package? – agenis Sep 02 '15 at 16:35

score 11 · Accepted Answer · answered Jul 26 '13 at 14:19

The short answer is that this was a bug in parallel backends such as doSNOW, doParallel and doMPI, but it has since been fixed.

The slightly longer answer is that foreach exports functions to the workers using a special "export" environment, not the global environment. That used to cause problems for functions that were created in the global environment, because the "export" environment wasn't in their scope, even though they were now defined in that same "export" environment. Thus, they couldn't see any other functions or variables defined in the "export" environment, such as "Posdef" in your case.

The doSNOW, doParallel and doMPI backends now change the associated environment from the global to the "export" environment for functions exported via ".export", and seems to have resolved these issues.

I thought this issue was solved, but I am still having the same problem with the packages `foreach v 1.4.4`,`doMPI v 0.2.2`, `doParallel v 1.0.14`, `doSNOW v 1.0.16`, running on `R v 3.5.0` with `RStudio v 1.1.453` — Nemesi, Feb 06 '19 at 10:47

score 1 · Answer 3 · answered Aug 21 '17 at 09:08

1

Quick fix for problem with foreach %dopar% is to reinstall these packages:

install.packages("doSNOW")

install.packages("doParallel") 

install.packages("doMPI")

It worked in my case.

answered Aug 21 '17 at 09:08

M_D

287
3
13

could not find function inside foreach loop

3 Answers3

Linked