Execution time do not decrease even after using parallel package in R?

Question

library(parallel)
# Calculate the number of cores
no_cores <- detectCores() - 1
# Initiate cluster
cl <- makeCluster(no_cores)

statesNames=c("a","b","c")
mcA<-new("markovchain", states=statesNames, transitionMatrix=matrix(c(0.2,0.5,0.3,0,0.2,0.8,0.1,0.8,0.1),nrow=3, byrow=TRUE, dimnames=list(statesNames,statesNames)))
mcB<-new("markovchain", states=statesNames, transitionMatrix=matrix(c(0.2,0.5,0.3,0,0.2,0.8,0.1,0.8,0.1),nrow=3, byrow=TRUE, dimnames=list(statesNames,statesNames)))
mcC<-new("markovchain", states=statesNames, transitionMatrix=matrix(c(0.2,0.5,0.3,0,0.2,0.8,0.1,0.8,0.1),nrow=3, byrow=TRUE, dimnames=list(statesNames,statesNames))) 
mclist <- new("markovchainList", markovchains = list(mcA, mcB, mcC))   
mc <- mclist

clusterExport(cl, "mclist")

f <- function(x) {
  n <- length(mclist@markovchains)
  seq <- character(length = n)

  t0  <- (mclist@markovchains[[1]]@states)[1]


  for(i in 1:n) {
    stateName <- mclist@markovchains[[i]]@states
    t0 <- sample(x = stateName, size = 1, 
           prob = mclist@markovchains[[i]]@transitionMatrix[which(stateName == t0 ), ]) 

    seq[i] <- t0
  } 
  return(seq)
}

I have two function which performs same task. One is using parallel package and another does not. I thought after using the parallel package the execution will be fast. But rather it seems to be slow.

> microbenchmark(rmarkovchain(100, mc, "matrix",useRCpp = F), parSapply(cl, 1:100,f))
Unit: milliseconds
                                         expr       min        lq      mean    median        uq      max neval
 rmarkovchain(100, mc, "matrix", useRCpp = F)  3.632955  4.251373  5.611569  5.507326  6.681284 11.92689   100
                      parSapply(cl, 1:100, f) 40.929350 43.893277 45.516566 45.373365 47.366842 52.80290   100

Since I am using linux I have used mclapply instead of parSapply and now its better than parSapply but still slower.

> microbenchmark(rmarkovchain(100, mc, "matrix",useRCpp = F), mclapply(cl, 1:100,f))
Unit: milliseconds
                                         expr       min       lq      mean    median        uq      max neval
 rmarkovchain(100, mc, "matrix", useRCpp = F)  3.798599  3.97889  6.636692  6.053313  8.935721 18.08281   100
                       mclapply(cl, 1:100, f) 14.862175 20.81366 26.211019 25.636895 31.893560 34.42886   100

Why I am not able to speed up the work using parallel package in R?

probably because `f` is a much slower function than `rmarkovchain` — Chris, May 25 '16 at 18:27
How many cores do you have? Often, the overhead of doing the parallel processing can be more 'expensive' than the speed up of distributing it. This is especially the case if the number of cores is small. — Andrew Taylor, May 25 '16 at 18:29
Possible duplicate of [How can I make a parallel operation faster than the serial version?](http://stackoverflow.com/questions/24398905/how-can-i-make-a-parallel-operation-faster-than-the-serial-version) — lmo, May 25 '16 at 18:38

score 0 · Answer 1 · edited May 23 '17 at 10:34

I totally agreed with Andrew Taylor.

I have read this post and there I read it will be useful to use parallel package when a certain task is taking long time(a few seconds).

To test this I increases the number of output i.e. from 100 to 10000

> microbenchmark(rmarkovchain(10000, mc, "matrix",useRCpp = F), mclapply(cl, 1:10000,f))
Unit: milliseconds
                                           expr        min         lq       mean     median         uq        max neval
 rmarkovchain(10000, mc, "matrix", useRCpp = F) 1140.07516 1186.11030 1294.44378 1224.73236 1401.15991 1824.34178   100
                       mclapply(cl, 1:10000, f)   18.27705   22.48832   28.23278   30.33396   33.06192   42.69159   100

Now parallel processing is faster than sequential execution.

still probably lots of room for optimization within your `f` function... subsetting within a loop is often a huge hangup, and can be vectorized — Chris, May 25 '16 at 19:05

Execution time do not decrease even after using parallel package in R?

1 Answers1