1

I am developing a parallel R code using the Snow package, but when calling C++ code using the Rcpp package the program just hangs and is unresponsive.

as an example... I have the following code in R that is using snow to split into certain number of processes

    MyRFunction<-function(i) {
      n=i
      .Call("CppFunction",n,PACKAGE="MyPackage")
      }
    if (mpi) {
      cl<-getMPIcluster()
      clusterExport(cl, list("set.user.Random.seed"))  
      clusterEvalQ(cl, {library(Rcpp); NULL})
      out<-clusterApply(cl,1:mc.cores,MyRFunction)
      stopCluster(cl)
      }
    else
      out <- parallel::mclapply(1:mc.cores,MyRFunction)

Whereas my C++ function looks like...

    RcppExport SEXP CppFunction(SEXP n) {
      int n=as<int>(n);
      }

If I run it with mpi=false and mc.cores=[some number of threads] the program runs beautifully BUT if i run it with mpi=true, therefore using snow, the program just hangs at int=as<int>(n) ????? On the other hand if I define the C++ function as...

    RcppExport SEXP CppFunction(SEXP n) {
      CharacterVector nn(n);
      int n=boost::lexical_cast<int>(nn[0]);
      }

The program runs perfectly on each mpi thread?? The problem is that it works for integers doubles etc, but not matrices Also, I must use lexical_cast from the boost package to make it works since as<> does not.

Does anybody know why this is, and what I am missing here, so I can load my matrices as well?

MenzZana
  • 158
  • 5

2 Answers2

2

It is not entirely clear from your question what you are doing but I'd recommend to

  • simplify: snow certainly works, and works with Rcpp as it does with other packages

  • trust packages: I found parallel computing setups easier when all nodes are identical local packages sets

  • be careful with threading: if you have trouble with explicit threading in the snow context, try it first without it and the add it once the basic mechanics work

Dirk Eddelbuettel
  • 360,940
  • 56
  • 644
  • 725
  • I guess it isn't as identical to the question I referenced as I thought since you answered that one too. – Dean MacGregor Jun 12 '14 at 21:25
  • Thanks for the suggestions. I have asked the question in the Rcpp mailing list and hopefully someone knows what I am doing wrong – MenzZana Jun 19 '14 at 12:37
  • Well, I am am the owner/admin of that list so I'd tell you the same there (but there are other knowledgeable subscribers). Your post has not come through though, so make sure you are subscribed. – Dirk Eddelbuettel Jun 19 '14 at 14:19
  • Thanks Dirk and have not posted on Rcpp yet, since waiting for confirmation to subscription, although as you can see above I have simplified the code so that it is clearer what the problem is. – MenzZana Jun 19 '14 at 19:34
  • There is no confirmation. You sign up first, then you can post. – Dirk Eddelbuettel Jun 19 '14 at 19:36
0

Finally the issue was resolved, and the problem seems to lie with getMPICluster() which works perfectly fine for pure R code, but not as well with Rcpp, as explained above. Instead using makeMPICluster command

    mc.cores <- max(1, NumberOfNodes*CoresPerNode-1) # minus one for master
    cl <- makeMPIcluster(mc.cores)
    cat(sprintf("Running with %d workers\n", length(cl)))
    clusterCall(cl, function() { library(MyPackage); NULL })
    out<-clusterApply(cl,1:mc.cores,MyRFunction)
    stopCluster(cl)

Works great! The problem is that you have to manually define the number of nodes and cores per node within the R code, instead of defining it using the mpirun command.

MenzZana
  • 158
  • 5