How can I evaluate a C function from a dynamic library in an R package?

Question

I’m trying to implement parallel computing in an R package that calls C from R with the .C function. It seems that the nodes of the cluster can’t access the dynamic library. I have made a parallel socket cluster, like this:

cl <- makeCluster(2)

I would like to evaluate a C function called valgrad from my R package on each of the nodes in my cluster using clusterEvalQ, from the R package parallel. However, my code is producing an error. I compile my package, but when I run

out <- clusterEvalQ(cl, cresults <- .C(C_valgrad, …))

where … represents the arguments in the C function valgrad. I get this error:

Error in checkForRemoteErrors(lapply(cl, recvResult)) : 
  2 nodes produced errors; first error: object 'C_valgrad' not found

I suspect there is a problem with clusterEvalQ’s ability to access the dynamic library. I attempted to fix this problem by loading the glmm package into the cluster using

clusterEvalQ(cl, library(glmm))

but that did not fix the problem.

I can evaluate valgrad on each of the clusters using the foreach function from the foreach R package, like this:

out <- foreach(1:no_cores) %dopar% {.C(C_valgrad, …)}

no_cores is the number of nodes in my cluster. However, this function doesn’t allow any of the results of the evaluation of valgrad to be accessed in any subsequent calculation on the cluster.

How can I either

(1) make the results of the evaluation of valgrad accessible for later calculations on the cluster or

(2) use clusterEvalQ to evaluate valgrad?

"I can evaluate valgrad on each of the clusters using the foreach function ..." - the reason for this is probably because you are on Unix/macOS and use `registerDoParallel(ncores)`, which creates **forked** processes. Using `cl <- makeCluster(ncores, type = "FORK")` would probably work equally well. That is, there's should be nothing magic with foreach or its backends here. The answer by Rui B explains what the problem is. — HenrikB, Oct 30 '18 at 04:48

Rui Barradas · Answer 1 · 2018-10-30T07:03:36.480

You have to load the external library. But this is not done with library calls, it's done with dyn.load.
The following two functions are usefull if you work with more than one operating system, they use the built-in variable .Platform$dynlib.ext.
Note also the unload function. You will need it if you develop a C functions library. If you change a C function before testing it the dynamic library has to be unloaded, then (the new version) reloaded.

See Writing R Extensions, file R-exts.pdf in the doc folder, section 5 or on CRAN.

dynLoad <- function(dynlib){
    dynlib <- paste(dynlib, .Platform$dynlib.ext, sep = "")
    dyn.load(dynlib)
}

dynUnload <- function(dynlib){
    dynlib <- paste(dynlib, .Platform$dynlib.ext, sep = "")
    dyn.unload(dynlib)
}

Does this still help me if I build the package? **Writing R Extensions** states `.C` provides "an interface to compiled code that has been linked into R, either at build time or via `dyn.load`" — bensonsyd, Oct 31 '18 at 02:12

How can I evaluate a C function from a dynamic library in an R package?

1 Answers1