0

I have the following (sample) code which is executed in passive mode on computing clusters:

> foo <- function(...)
> {
> require(copula)
> require(foreach)
> require(methods)
> require(np)
> require(parallel)
> {cl <- makeCluster(detectCores()-1) ; registerDoParallel(cl)}
...
> str(dat)
'data.frame':   31269 obs. of  15 variables
...
> clusterExport(cl=cl,varlist=c("dat",...))
> dumpster <- foreach(i=1:10,.packages=c("copula","foreach","np"),.export=c("dat"))%dopar%
> {... ; dat.new <- some.fn(dat) ; ...}
...
> stopCluster(cl)
> }
> foo(...)
Error in get(name, envir = envir) : object 'dat' not found
Calls: foo ... postNode -> sendData -> sendData.SOCKnode -> serialize -> get
Execution halted

What am I doing wrong?

Cœur
  • 37,241
  • 25
  • 195
  • 267
DeeCeeDelux
  • 105
  • 7

1 Answers1

0

You have mixed up some instruction, to make order do this

library(doSNOW)
library(foreach)
cl <- makeCluster(detectCores()-1, type = "SOCK")
registerDoSNOW(cl)

I'm assuming that the object dat was already availaible and so this is how you use the foreach

dumpster <- foreach(i=1:10,.packages=c("copula","np"), .combine = c)%dopar%{dat.new <- some.fn(dat)}

Then close the clusters

stopCluster(cl)
tia_0
  • 412
  • 1
  • 3
  • 11
  • Thanks for your answer. But indeed there was nothing mixed up. I chose the 'parallel' package since it suited my needs. The other packages are needed for my computations and I posted them in case some incompatibilities existed without my knowledge. Finally, the reason was that 'dat' was too big to be replicated so often within the cluster, causing memory issues and hence the cluster to break. – DeeCeeDelux Aug 02 '16 at 20:36
  • I had the same problem myself, I hope all worked out for you. – tia_0 Aug 02 '16 at 20:38
  • Yeah finally... Just a pain in the ass to find out based on the "object 'dat' not found" error which - in my eyes - makes no sense in case of such an error. – DeeCeeDelux Aug 02 '16 at 20:41
  • Exactly, I thought that something went wrong in the environment and so R wasn't able to find the object while parallelizing the function. – tia_0 Aug 02 '16 at 20:43