2

I'm working with a custom random forest function that requires both a starting and ending point in a set of genomic data (about 56k columns).

I'd like to split the column numbers into subgroups and allow each subgroup to be processed individually to speed things up. I tried this (unsuccessfully) with the following code:

library(foreach)
library(doMC)
foreach(startMrk=(markers$start), endMrk=(markers$end)) %dopar% 
    rfFunction(genoA,genoB,0.8,ntree=100,startMrk=startMrk,endMrk=endMrk)

Where startMrk is an array of numeric variables: 1 4 8 12 16 and endMrk is another array: 3 7 11 15 19

For this example, I'd want one core to run samples 1:3, another to run 4:7, etc. I'm new to the idea of parallel processing in R, so I'm more than willing to study any documentation available. Does anyone have advice on things I'm missing for parallel-wise processing or for the above code?

Steve Weston
  • 19,197
  • 4
  • 59
  • 75
trvrr
  • 25
  • 5

1 Answers1

1

The basic point here is that you're splitting up your columns into chunks, right. First, it might be better to chunk your dataset appropriately at each iteration and feed the chunks into RF. Also, foreach works just like for in some ways, so the code can be

rfs=vector('list',4)
foreach(i=1:4) %dopar% {
  ind <- markers$start[i]:markers$end[i]
  rfs[[i]] <- randomForest(genoA[,ind],genoB[,ind], 0.8, ntree=100)
}

I gave this in regular randomForest, but you can wrap this up into your custom code in a straightforward manner.

Abhijit
  • 625
  • 1
  • 5
  • 9
  • That didn't work exactly right, but when I used something like: `r <- foreach(i=1:5, .combine="cbind") %dopar%{ RandomForest(genoA,genoB,0.8,ntree=100,startMrk=markers$start[i],endMrk=markers$end[i])}` I got back what I was wanting. One final question: the custom function I'm using returns a list. Is there a way to selectively combine elements from the list? Thanks for your help! – trvrr Feb 12 '13 at 18:01
  • A little late, but to answer your "final question": Rather than "selectively combining elements from a list", why not just extract the list element you're interested in? Rather than: `foreach(...) %dopar% {return(list(A = ..., B = ...))}`; try: `foreach(...) %dopar% {res <- list(A = ..., B = ...); return(res$A)}`. Alternatively, define your own `.combine` function which does the same thing. (The first option will result in less data moving around though; important if `res$B` is large!) – dynamo Jan 23 '14 at 09:50