2

I am writing an R package and would like to equip one of the functions with the option to run in parallel. Based on the advice from this response, I would like to give the user the option to either register their own parallel backend, or allow the function to do it for them automatically if it does not detect any registered backend. As mentioned in this response, it is also good practice to give the user the option to stop the implicitly created cluster.

In implementing this, I have noticed that it is possible for the workers to be registered, but not on an active cluster. That is, foreach::getDoParRegistered() may return TRUE, but attempting a parallel operation will give an error. For example:

> doParallel::registerDoParallel(cores = 3)
> foreach::getDoParRegistered()
[1] TRUE
> foreach::`%dopar%`(foreach::foreach(j = 1:3), Sys.sleep(1))
[[1]]
NULL

[[2]]
NULL

[[3]]
NULL

> doParallel::stopImplicitCluster()
> foreach::getDoParRegistered()
[1] TRUE
> foreach::`%dopar%`(foreach::foreach(j = 1:3), Sys.sleep(1))
Error in summary.connection(connection) : invalid connection

Thus, it seems that the code should not only check for registered workers, but also if these workers are associated with an active cluster. I have developed the following MWE to demonstrate my approach:

Restarting R session...

par_sleep <- function(stop_cluster = T){

  registered <- tryCatch(
    {
      # return true if parallel backend is registered with more than 1 worker
      foreach::`%dopar%`(foreach::foreach(NULL), NULL)
      foreach::getDoParRegistered()
      },

    # return false if error
    error = function(msg) F,

    # return false if warning
    warning = function(msg) F)

  if (!registered){
    warning("Active parallel backend not detected; registering parallel backend using 3 cores")
    doParallel::registerDoParallel(cores = 3)
  } else{
    message(paste("Detected", foreach::getDoParWorkers(), "registered workers on an active cluster"))
  }
  start <- Sys.time()
  foreach::`%dopar%`(foreach::foreach(j = 1:3), Sys.sleep(1))
  print(Sys.time() - start)
  if(stop_cluster){
    doParallel::stopImplicitCluster()
  }
}


> par_sleep(F)
Time difference of 1.105459 secs
Warning message:
In par_sleep(F) :
  Active parallel backend not detected; registering parallel backend using 3 cores
> par_sleep()
Detected 3 registered workers on an active cluster
Time difference of 1.144499 secs
> par_sleep(F)
Time difference of 1.104962 secs
Warning message:
In par_sleep(F) :
  Active parallel backend not detected; registering parallel backend using 3 cores

I am wondering if this is a good approach to offering the user the option of automated backend registration, and if there are any other considerations I am missing.

Further, based on this post, it seems that using the cores argument in my call to doParallel::registerDoParallel is more flexible than the cl argument in terms of its behavior on different operating systems. Is this the correct conclusion? For my use case, is it better to use the cores argument?

Finally, I am allowing the user to specify the number of workers that they would like for the cores argument. To try to verify this is a valid number, I check that it is positive and an integer. Provided that parallel::detectCores() does not return NA, I also check that the number of workers specified is not greater than the number of detected cores. However, I am not sure if there is a case where detectCores could return a number less than the number of workers; if so, I would like to switch this error message to a warning.

Jacob Helwig
  • 41
  • 1
  • 5
  • That sounds like a bug in `doParallel::stopImplicitCluster`, I think it should call `foreach::registerDoSEQ`. I personally wouldn't even implement an option where the function itself creates the parallel workers, I would always leave it to the user, but that's just my opinion. – Alexis Feb 11 '22 at 22:30

0 Answers0