0

Any help of this would be greatly appreciated.

I have the following R code (following tutorial at https://www.r-bloggers.com/parallel-r-loops-for-windows-and-linux/) to carry out 233 mutually exclusive calculations in parallel. There are 4 cores and 32 GB of RAM on the machine where this code is executing. 

# Calculate the number of cores
    no_cores <- detectCores() - 1
    # initiate cluster
    cl <- makeCluster(no_cores)
    registerDoSNOW(cl)

    # Process sandboxes in parallel - there are 233 sandboxes in total
    foreach (j=1:nrow(sandboxes), .packages=c("dplyr", "rJava", "RJDBC", "lazyeval", "caTools", "stringr", "readxl")) %dopar%
    {
      print("Execution is in foreach block now")
      // Do stuff
    }
    stopCluster(cl)

I am getting the error message even before the execution starts inside the foreach loop. 

Error in { : 
  task 3 failed - "RcallMethod: attempt to call a method of a NULL object."

Here is the output from sessionInfo()

R version 3.3.3 (2017-03-06)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 7 x64 (build 7601) Service Pack 1

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252    LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C                           LC_TIME=English_United States.1252    

attached base packages:
[1] parallel  stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] doSNOW_1.0.14     snow_0.4-2        doParallel_1.0.10 iterators_1.0.8   foreach_1.4.3     readxl_0.1.1      stringr_1.2.0     caTools_1.17.1   
 [9] lazyeval_0.2.0    RJDBC_0.2-5       DBI_0.6-1         rJava_0.9-8       dplyr_0.5.0      

loaded via a namespace (and not attached):
 [1] Rcpp_0.12.10     codetools_0.2-15 assertthat_0.1   bitops_1.0-6     R6_2.2.0         magrittr_1.5     stringi_1.1.3    tools_3.3.3     
 [9] compiler_3.3.3   tibble_1.3.0
KalC
  • 1,530
  • 3
  • 22
  • 33
  • 1
    Are you exporting your data so that are made available to the workers? Look up for `?clusterExport()`. – Damiano Fantini Aug 28 '17 at 17:50
  • From https://stackoverflow.com/questions/18028452/reading-global-variables-using-foreach-in-r ... "The doParallel package will auto-export variables to the workers that are referenced in the foreach loop." - No? Do I need to explicitly export all the data? – KalC Aug 28 '17 at 17:51
  • 3
    The problem is in your "do stuff" and we know nothing about that. Have you tried running the loop with %do%? – Roland Aug 28 '17 at 17:54
  • Yes, it's kinda difficult to spot the problem here... – Damiano Fantini Aug 28 '17 at 17:54
  • @Roland - that is indeed the case. I put in something simple like print(j) and it worked. Will look at clusterExport() – KalC Aug 28 '17 at 18:00
  • Hello, have you solved the problem? I am facing with the same issue. I used clusterExport but it didn't work. – boyaronur Jan 15 '19 at 08:28
  • Unfortunately no. I used data.table (instead of data.frames) to overcome some of the challenges. Parallel execution is only good for CPU intensive tasks. Mine was memory intensive. Sorry :( – KalC Jan 15 '19 at 16:54

0 Answers0