Snowfall's sfApply and sfClusterApplyLB is slower than normal loop or sapply

Question

When i apply this code in R, the loop and sapply are faster than snowfall's functions. What am i doing wrong? (using windows 8)

library(snowfall)
a<- 2
sfInit(parallel = TRUE, cpus = 4)
wrapper <- function(x){((x*a)^2)/3}
sfExport('a')
values <- seq(0, 100,1)
benchmark(for(i in 1:length(values)){wrapper(i)},sapply(values,wrapper),sfLapply(values, wrapper),sfClusterApplyLB(values, wrapper))
sfStop()

elapsed time for after 100 replications:

loop              0.05
sapply            0.07
sfClusterApplySB  2.94
sfApply           0.26

score 1 · Answer 1 · edited May 23 '17 at 11:57

If the function that is sent to each of the worker nodes takes a small amount of time, the overhead of paralellization causes the overall duration of the task to take longer than running the job serially. When the jobs that are sent to the worker nodes take a significant amount of time (at least several seconds), than paralellization will really show improved performance.

Snowfall's sfApply and sfClusterApplyLB is slower than normal loop or sapply

1 Answers1