R parallel processing with nested loops

Question

I am trying to increase the execution speed of the code below, but I am not sure whether to parallelize the outermost loop only or the outer loop and inner loops. I am working on Ubuntu with 2 processors and I do not know how many threads each processor would create to conduct this task and whether the spanning of many threads would bring any complications that I should be aware of and control with locks. What would you recommend?

ibrary(foreach)
library(doParallel)

nc = detectCores()
cl = makeCluster(nc, typr = “FORK”)
registerDoParallel(cl)

pts <- list(chunkSize=2)

    foreach (H in 0:HexC, .combine = “c”) %:%{
        foreach (HN in 0:HNcC,  .Combine = “c”) %dopar%{
            foreach (F in 0:FucC, .Combine = “c” ) %dopar%{
                foreach (SA in 0:SAC, .Combine = “c”) %dopar%
                    foreach (SO3 in 0:SO3C,{
                        NAmax<- sum(SA+SO3)
                        foreach (NAD in 0:NAmax, .combine = “c”) %dopar%{
                            Na_Cnt<- c(Na_Cnt, NAD)
                            SO3_Cnt<- c(SO3_Cnt, SO3)
                            SA_Cnt<- c(SA_Cnt, SA)
                            Fuc_Cnt<- c(Fuc_Cnt, F)
                            HexNAc_Cnt<- c(HexNAc_Cnt, HN)
                            Hex_Cnt<- c(Hex_Cnt, H)

                            Na_Mass<- c(Na_Mass, NAD*NaAdductMass)
                            SO3_Mass<- c(SO3_Mass, SO3*dels["SO3"])
                            SA_Mass<- c(SA_Mass, SA*dels["SA"])
                            Fuc_Mass<- c(Fuc_Mass, F*dels["Fuc"])
                            HexNAc_Mass<- c(HexNAc_Mass, HN*dels["HexNAc"])
                            Hex_Mass<- c(Hex_Mass, H*dels["Hex"])
                        }
                    }
                }
            }
        }
    }

stopImplicitCluster()
stopCluster(cl)

Without any data or description of what the above code is suppose to do, I can't provide a full answer. This code needs to be optimized first, before trying to apply parallelization. This should be able to be vectorized to remove a few of the nested loops. — Dave2e, May 07 '18 at 17:07
The code is creating vectors to insert into a dataframe and later on write the dataframe to a file. bigTab<- data.frame(Hex_Cnt, HexNAc_Cnt, Fuc_Cnt, SA_Cnt, SO3_Cnt, Na_Cnt, Hex_Mass, HexNAc_Mass, Fuc_Mass, SA_Mass, SO3_Mass, Na_Mass). it uses dels<- c( 162.0528, 203.0794, 146.0579, 291.0954, 79.9568) names(dels)<-c( "Hex", "HexNAc", "Fuc", "SA", "SO3") NaAdductMass<- xx.0000 — gccd, May 07 '18 at 17:25
These are the parameters received by the function. They are constant values: bigTabFn, HexC, HNcC, FucC, SAC, SO3C — gccd, May 07 '18 at 17:31
The reason why I am trying to parallelize the code is the following: These statements belong in the middle of the program. There are many more processes and statements that happen before the program gets to the nested loops. If I remove them from the code, all the processes and statements run in order. When I introduce this code, the program starts running from the nested loops and the previous statements are omitted. The 2 processors are super busy. Additionally, running the nested loops takes a large a mount of time — gccd, May 07 '18 at 17:37
My gut feel on this one is that parallelization may not give you the most bang for your buck here. If your problem is inescably serial, Rppp might be helpful, but most likely using basic vectorization techniques in R will give you the most improvement for your time. [Example 1](https://stackoverflow.com/questions/49136712/r-for-loop-within-a-foreach-loop/) - [Example 2](https://stackoverflow.com/questions/49177163/parallelization-over-for-loop-analyzing-a-data-frame/). Can you add a self-contained example providing a minimal set of inputs, expected outputs, and non-parallelized code? — Matt Summersgill, May 07 '18 at 19:09

Katia · Accepted Answer · 2018-05-07T19:42:07.220

There are multiple problems with your code. You code calculate the output variable based on the value of that variable in the previous iteration, for example:

Na_Mass<- c(Na_Mass, NAD*NaAdductMass)

Instead you need to do something like this:

Na_Mass <- foreach (NAD in 0:NAmax, .combine = “c”) %dopar%{
   return(NAD*NaAdductMas)

}

For more examples see the documentation of doParallel packages: https://cran.r-project.org/web/packages/doParallel/vignettes/gettingstartedParallel.pdf

As for the number of CPU cores your program will use, it will be equal to HNcC * FucC * SAC * NAmax, which is probably a very large number and with your computer having only 2 processors you will run in danger of burning it. And at the same time each parallel process in R will not have enough CPU resources and will run significantly slower. I would parallelize no more than one loop.

One more note: This approach to calculate Na_cnt, and other objects in your loop is VERY slow with or without parallelization:

Na_Cnt<- c(Na_Cnt, NAD)

Instead you should vectorize:

Na_Cnt <- 0:NAmax

Similarly:

  SO3_Cnt<- rep( SO3, NAMax+1)
  SA_Cnt<- rep( SA, NAMax+1)
  Fuc_Cnt <- rep(F, NAMax+1)
  HexNAc_Cnt <- rep( HN, NAMax+1)
  Hex_Cnt <- rep(H,NAMax+1 )

Similar with all other statements in your innermost loop. This will be significantly faster and you will not need any parallelization.

You are welcome! If this answers your question, please click "accept" answer. Thank you. — Katia, May 08 '18 at 18:24
Bellow the arrows to the left of the answer text, there is a gray check. When you click it it will make the answer marked "accepted". Thank you. — Katia, May 08 '18 at 21:10

R parallel processing with nested loops

1 Answers1