1

As recommended in other posts I wrote my own package in R to parallelize functions I wrote with Rcpp. I can load the package and everything works, but when I'm using optimParallel, I get the message:

Error in checkForRemoteErrors(val) : 3 nodes produced errors; first error: object '_EffES_profileLLcpp' not found

Here is what I'm doing: 

library(optimParallel)
library(EffES) # EffES is my own package

cl <- makeCluster(detectCores()-1)
clusterEvalQ(cl, library(EffES))
clusterEvalQ(cl, library(optimParallel))
setDefaultCluster(cl = cl)

w.es <- optimParallel(par=rep(0.001,3), profileLLcpp, y=y.test, x=x.test, lower = rep(0.001,3), method = "L-BFGS-B")$par

Error in checkForRemoteErrors(val) : 
  3 nodes produced errors; first error: object '_EffES_profileLLcpp' not found

What am I doing wrong?

To Mate
  • 51
  • 6
  • Log into the worker node, try loading it and see what happens. Maybe the `.libPaths()` is different. Maybe you forgot to install `EffES` on one or more nodes. We can't tell. – Dirk Eddelbuettel Sep 25 '18 at 16:33
  • How many cores does your computer have? Which tools do you use for package development? – Ralf Stubner Sep 25 '18 at 19:06
  • My laptop has 4 cores. I used this instrution (section 4) to build my package: [link](http://web.mit.edu/insong/www/pdf/rpackage_instructions.pdf). I used the functions `RcppArmadillo.package.skeleton()`, `compileAttributes()` and `package_native_routine_registration_skeleton()` from RcppArmadillo, Rcpp and tools. And then I built and installed it in the commander with `Rcmd build` and `Rcmd INSTALL` – To Mate Sep 26 '18 at 06:08
  • Is your package source code publicly available, for example in a GitHub repo? – duckmayr Sep 26 '18 at 10:17
  • 1
    It's not in a GitHub repo, but maybe this helps:[src](https://codebunk.com/b/515249891/) – To Mate Sep 26 '18 at 10:59
  • 2
    The codebunk link does not help much, since it only contains your "business logic". Here all the glue around that file are of interest. For example, I don't think that there is a need to use `tools::package_native_routine_registration_skeleton()` since `Rcpp::compileAttributes()` already does that. – Ralf Stubner Sep 26 '18 at 11:48
  • 2
    And by the same token, if a "complicated" or "non-standard" package fails, try a simpler one. My first test for distributed work always is to report the process ID, or hardware IP address, or ... back. – Dirk Eddelbuettel Sep 26 '18 at 12:38

2 Answers2

0

You have to spread the object '_EffES_profileLLcpp' to each core of your cluster. You can do this using clusterExport, in your case: 

clusterExport(cl,'_EffES_profileLLcpp')

Repeat this step with every object needed to be used in parallel (or just check which object shows up in the error log and spreat it using clusterExport).

Hope this helps

antonioACR1
  • 1,303
  • 2
  • 15
  • 28
  • One doesn't need multiple lines. `clusterExport` has `varlist` parameter, so you can specify multiple exports in a single line of code – alexwhitworth Oct 12 '18 at 03:49
  • True, although maybe you don't know which objects you need to spread in the first place (or at least that's what happened to me because I was trying to run some long and complicated code) and the error indicates which objects need to be spread. – antonioACR1 Oct 12 '18 at 15:23
0

Edit: The problem is solved in optimParallel version 0.7-4

The version is available on CRAN: https://CRAN.R-project.org/package=optimParallel

For older versions:

As detailed in this post optimParallel() needs to trick a bit in order to have no restrictions on the argument names that can be passed through the ... argument. Currently, this implies that the function passed to optimParallel() has to be defined in the .GlobalEnv in order to find compiled code properly.

Hence, a workaround could be to define the function in the .GlobalEnv:

library(optimParallel)
library(EffES)                          # EffES is your own package
cl <- makeCluster(detectCores()-1)
clusterEvalQ(cl, library(EffES))
setDefaultCluster(cl=cl)

f <- function(par, y, x) {
    profileLLcpp(par=par, x=x, y=y)
}
optimParallel(par=rep(0.001,3), f=f, y=y.test, x=x.test, 
              lower = rep(0.001,3), method = "L-BFGS-B")$par

Suggestions to improve the code of optimParallel() are welcome. I opened a corresponding question here.

Nairolf
  • 2,418
  • 20
  • 34