4

I am using R with Rcpp to perform computationally expensive calculations which also require a lot of RAM. Since I often do these for different parameters, I want to calculate them in parallel. For this I use packages foreach and doParallel. My problem is that once a worker on a thread has finished, it seems like it does not release the RAM. For example, if I use 7 cores and want to scan 9 parameters, I get approximately this behavior: enter image description here

You see that the jump in memory is roughly the same for the two threads 8 and 9 as it is for the seven threads 1-7. Only after 8 and 9 the memory seems to be released.

My minimal working example in R:

library(minpack.lm)
library(Rcpp)
library(myRcppPackage)
library(foreach)
library(doParallel)

myParameters <- c(1:9)

# setup parallel backend
cores=detectCores()
close( file( "./monitorfile.txt", open="w" ) ) # flush the monitor-file
cl <- makeCluster(cores[1]-1, outfile="./monitorfile.txt")
registerDoParallel(cl)

clusterExport(cl, list("performCppLoop"), envir = environment())

myResult <- foreach(i=1:length(myParameters), .combine=rbind) %dopar% {

  # perform C++ loop with my parameters
  myData <- performCppLoop(myParameters[i])

  # do some stuff with myData
  cbind(mean(myData[,1]), mean(myData[,2]), mean(myData[,3]))  

  rm(myData)
} 

stopCluster(cl)

MWE C++ code:

  // [[Rcpp::export]]
NumericMatrix performLoop(double myParameter){

    const number_of_steps = 20000000;

     // ps for phase space
    NumericMatrix data(number_of_steps, 3);

    for (unsigned long int i = 0; i <number_of_steps; i++) {
        // just some dummy calculations
        data(i, 0) = sqrt(myParameter);
        data(i, 1) = myParameter*2.0;
        data(i, 2) = myParameter/2.0;
    } 

    return data;
}

What am I doing wrong?

Christian
  • 169
  • 7
  • 3
    Try gc() and rm(): What is the difference between gc() and rm() https://stackoverflow.com/questions/8813753/what-is-the-difference-between-gc-and-rm – HelloWorld Apr 30 '20 at 07:17
  • 1
    That has nothing to do with Rcpp, but a lot with how R works. – Dirk Eddelbuettel Apr 30 '20 at 11:31
  • 2
    @HelloWorld That fixes it, thank you. Do you want to post it as answer? – Christian Apr 30 '20 at 12:06
  • 2
    @DirkEddelbuettel I was not sure where the error comes from and I am by no means an expert in R or C++. Just an experimental physicist who is forced to code during lockdown ;) – Christian Apr 30 '20 at 12:06
  • I'm grad that I was able to help you! As much as I'd love to post it as an answer, somehow the stack overflow is posts my answer as a comment the every time I try. Unfortunately I have to pass this one. – HelloWorld Apr 30 '20 at 13:10

0 Answers0