4

I am trying to recruit more cores to increases my processing time for some lidar data I am analyzing but I keep getting "Error in makePSOCKcluster(names = spec, ...) : Cluster setup failed. 3 of 3 workers failed to connect." after I run this:

 UseCores <-detectCores() -1
cl <- makeCluster(UseCores)
registerDoParallel(cl) 
foreach(i=1:lengthcanopy_list)) %dopar% {
  library(raster)
  ttops <- vwf(CHM = canopy_test, winFun = lin, minHeight = 2, maxWinDiameter = NULL)
}

Why am I getting this error and what can I do to fix it?

lavp_theOG
  • 81
  • 1
  • 5
  • [This answer](https://stackoverflow.com/questions/17966055/makecluster-function-in-r-snow-hangs-indefinitely) could provide you something useful to try. – Trung Anh Phạm Jul 04 '20 at 18:08
  • notice that `length` has a missing parenthesis. Also, do not load libraries inside a parallelized code and use `FORK` instead or the `::` call to the package function. – Garini Oct 21 '20 at 20:39

2 Answers2

5

It seems a problem relative to recent versions of R. Until further updates, looking at this issue on GitHub it seems there are two workarounds as follows.

Directly use this to create the cluster:

cl <- parallel::makeCluster(2, setup_strategy = "sequential")

Or for a long term solution add the following to your ~/.Rprofile

## WORKAROUND: https://github.com/rstudio/rstudio/issues/6692
## Revert to 'sequential' setup of PSOCK cluster in RStudio Console on macOS and R 4.0.0
if (Sys.getenv("RSTUDIO") == "1" && !nzchar(Sys.getenv("RSTUDIO_TERM")) && 
    Sys.info()["sysname"] == "Darwin" && getRversion() >= "4.0.0") {
  parallel:::setDefaultClusterOptions(setup_strategy = "sequential")
}

Even if this workaround was necessary for Rstudio users, it could be of general use, as it is useful also on my GitLab registered runner tests.

Garini
  • 1,088
  • 16
  • 29
  • It's worth noting that this disables parallel processing. It's a workaround in the sense that the code doesn't hang, and doesn't require extensive code surgery to switch from parallel to sequential processing. But it's not a workaround in the sense of "running in parallel despite the bug with RStudio." – Dan Hicks Feb 24 '21 at 17:06
  • 3
    This is not only OSX related. I had the same issue on Manjaro Linux and as you suggested, running the `parallel:::setDefaultClusterOptions(setup_strategy = "sequential")` in my session solved the issue. – Mehrad Mahmoudian Feb 24 '21 at 17:39
  • 3
    @DanHicks it runs in parallel, it is just creating the sockets sequentially. I just used it few minutes ago and it worked fine. – Mehrad Mahmoudian Feb 24 '21 at 17:40
2

Assuming you're working in RStudio, the problem was most likely this bug: changes in parallel created a conflict between R 4.0 and RStudio 1.3.something. Your code isn't a minimal working example so I can't check it on my end, but I just confirmed that cl = makeCluster(1) behaves as expected in RStudio 1.4.1103, R 4.0.2, Mac OS 10.15.7. So try updating RStudio and checking your code again.

Dan Hicks
  • 223
  • 1
  • 11