0

I am currently using foreach to do a relatively small task for many instances in parallel. I understand that foreach has to dispatch the processing which adds additional time to setup as well as combining the results. In my case, I do not need to combine any data but need fast parallel processing as the task is very sensitive to response times and is very short in duration. Because of the number of instances required to run, it is faster overall using parallel processing (doSNOW). Is there any alternative to make the parallel processing "setup" faster? Or perhaps, pre-staging somehow? I don't think this is possible but wanted to check with the R gurus first.

John Richardson
  • 676
  • 8
  • 24
  • 1
    Probably more context is needed; doSNOW spawns completely new processes and may require significant overhead, e.g., loading packages in the new processes. On the other hand, on non-Windows 'multicore' forks processes (marks existing memory as 'copy-on-right'), and is relatively inexpensive. Is your current implementation vectorized in a way that makes R processing efficient? This can easily gain 10-100x performance benefit, to typical parallel benefit of 2-5x. – Martin Morgan Nov 30 '14 at 02:32
  • Another option would be to write the bottle neck in C(++). The latter being efficiently implemented in goodness that is called `Rcpp`. – Roman Luštrik Nov 30 '14 at 10:23

0 Answers0