0

I'm facing a problem of performance when I tried to parallelise a calcul with R. I have a matrix of 8 columns (conditions) with 12000 row (genes), I would like to determine the optimal number of clusters representing the gene expression (group the genes with the same pattern of expression). To do that, I follow this Tutorial with the clustGap function and a partition around medoid.

As the calcul is pretty long, and as I can access to a cluster of calcul, I plan to parallelise it.

I would like to use snow package, and to evaluate the speed I extract a sub-matrix and I made a first test on my computer. 

library(snow)
cl<-makeCluster(8) 
clusterEvalQ(cl, library(cluster))
clusterExport(cl,"df")
T1<-Sys.time()
results <-clusterCall(cl,function(x) clusGap(df, FUN = pam, K.max = 20, B= 500,verbose=TRUE))
T2<-Sys.time() 
difftime(T2, T1)

Time difference of 14.59781 secs

T3<-Sys.time()
clusGap(df, FUN = pam, K.max = 20, B = 500,verbose=TRUE)
T4<-Sys.time()
difftime(T4, T3)

Time difference of 8.251367 secs

So I a little bit stuck by the test as it seems that the 1 core calcul is more efficient that the 8 cores :o

Does anybody know what I'm miss-understand in this calcul ?

Thanks a lot, 

sessionInfo()

> R version 3.3.3 (2017-03-06)
Platform: x86_64-apple-darwin13.4.0 (64-bit)
Running under: macOS  10.13.4

locale:
[1] fr_FR.UTF-8/fr_FR.UTF-8/fr_FR.UTF-8/C/fr_FR.UTF-8/fr_FR.UTF-8

attached base packages:
 [1] splines   stats4    parallel  stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] snow_0.4-2          cluster_2.0.6       DDRTree_0.1.5       irlba_2.3.1         VGAM_1.0-3          ggplot2_2.2.1      
[7] Biobase_2.34.0      BiocGenerics_0.20.0 Matrix_1.2-12      

loaded via a namespace (and not attached):
 [1] Rcpp_0.12.17           lattice_0.20-35        tidyr_0.8.1            GO.db_3.4.0            assertthat_0.2.0      
 [6] digest_0.6.15          slam_0.1-40            R6_2.2.2               plyr_1.8.4             RSQLite_2.1.1         
[11] pillar_1.2.3           rlang_0.2.1            lazyeval_0.2.1         data.table_1.11.4      blob_1.1.1            
[16] S4Vectors_0.12.2       combinat_0.0-8         qvalue_2.6.0           BiocParallel_1.8.2     stringr_1.3.1         
[21] igraph_1.1.2           pheatmap_1.0.10        bit_1.1-14             munsell_0.5.0          fgsea_1.0.2           
[26] pkgconfig_2.0.1        tidyselect_0.2.4       tibble_1.4.2           gridExtra_2.3          matrixStats_0.53.1    
[31] IRanges_2.8.2          dplyr_0.7.5            grid_3.3.3             gtable_0.2.0           DBI_1.0.0             
[36] magrittr_1.5           scales_0.5.0           stringi_1.2.3          GOSemSim_2.0.4         reshape2_1.4.3        
[41] bindrcpp_0.2.2         limma_3.30.13          DO.db_2.9              clusterProfiler_3.2.14 fastmatch_1.1-0       
[46] fastICA_1.2-1          RColorBrewer_1.1-2     tools_3.3.3            bit64_0.9-7            glue_1.2.0            
[51] purrr_0.2.5            HSMMSingleCell_0.108.0 AnnotationDbi_1.36.2   colorspace_1.3-2       DOSE_3.0.10           
[56] memoise_1.1.0          bindr_0.1.1
Ralf Stubner
  • 26,263
  • 3
  • 40
  • 75
Nicolas
  • 23
  • 6
  • Possible duplicate of [Why is the parallel package slower than just using apply?](https://stackoverflow.com/questions/14614306/why-is-the-parallel-package-slower-than-just-using-apply) – divibisan Jun 25 '18 at 20:44
  • Hi, thanks for the answer, If I understand well the post, authors said that gain of time is linked with the complexity of the calcul. So i tried to increase the size of my sub-matrix: `sub-matrix<-matrix[1:100,]` `difftime(T2, T1)` **Time difference of 11.29684 mins** `difftime(T4, T3)` **Time difference of 4.73247 mins** So It seems that even if I increase the complexity, I have this difference! – Nicolas Jun 25 '18 at 21:19
  • Well, the idea is that setting up each parallel calculation has a time cost. So if each individual calculation takes a short time to run, any gain from parallelization will be outweighed by the extra work to set up and merge the parallel calculations. So, while you might not see any benefit for a rapid calculation like this one (8 seconds is not a long time) you might see improvement if the calculation took several minutes or longer to run. – divibisan Jun 25 '18 at 21:32
  • Also, to gain a benefit from parallelization, functions must be written to make use of multiple cores. These links describe a `do.parallel=` argument in `clusGap` and an attempt to modify `clusGap` to allow parallel processing: https://rdrr.io/github/pimentel/cluster/man/clusGap.html, https://www.biostars.org/p/273107/ – divibisan Jun 25 '18 at 21:34
  • Try using `makeSOCKcluster` and never use all available threads (keep at least one). – F. Privé Jun 26 '18 at 06:21

0 Answers0