doParallel seems to corrupts output for unknown reason

Question

I'm attempting to run the following code (used for EWAS purposes) in a parallel format:

registerDoParallel(2)

combined <- foreach(i = 28:78, .combine=rbind)  %dopar%                 
{
mm<-rep(NA,412) 

FIDunique<-unique(pheno$FID)
for(j in FIDunique)
{
  nn<-which(pheno1$FID==j)
  mm[nn]<-mean(pheno1[nn,i])
}
tt4<-which(!is.na(mm))

if(length(tt4)>19 & length(table(pheno1[,i]))>1)
{
  meth<-pheno1[tt4,i]
  mm<-mm[tt4]

  if(length(levels(factor(pheno1$smoking)))<2) 
  {
    lme2<-tryCatch(lme(I(meth-mm)~delta_residSpine+I(DNA_year-AGE_year)+CD8T+CD4T+NK+Bcell+Mono+Gran,random=~1|FID, control=lmeControl(msMaxIter=50000,opt="optim"), data=pheno1[tt4,]),error = function(e) rep (NaN,1)) 
  }
  else 
  {
    lme2<-tryCatch(lme(I(meth-mm)~delta_residSpine+I(DNA_year-AGE_year)+factor(smoking)+CD8T+CD4T+NK+Bcell+Mono+Gran,random=~1|FID, control=lmeControl(msMaxIter=50000,opt="optim"), data=pheno1[tt4,]),error = function(e) rep (NaN,1))
  }

  if(!is.na(lme2)) out18[i-27,2:5]<-summary(lme2)$tTable[2,-3]

  rm(mm,nn,meth,lme2)
}

out18[i-27,6]<-length(tt4)
out18[i-27,]
}

The issue i'm having is that when I use %do% in the second line, the program executes perfectly and 'combined' contains the the correct output.

However when I use %dopar% (as is shown in the code above), the middle four columns of combined dissapear. I suppose this is due to those four columns being assigned based on a tryCatch? I've been stuck on this for a few days and really need to make progress.

It's also worth nothing that when I run the program as a %do%, I get the warning

Warning in if (!is.na(lme2)) out18[i - 27, 2:5] <- summary(lme2)$tTable[2, : the condition has length > 1 and only the first element will be used

for every iteration, however I do not get this at all when running under %dopar%.

Apologies if I haven't given enough information on what the variables are, i'm trying to disclose as little as possible about the actual information being processed for ethical reasons.

it's not easy to follow your problem. What Os are you running on ? `%do%` and `%dopar%` definitely don't have the same behaviour, and `%dopar%` can depend on the Os. — ClementWalter, Apr 20 '17 at 15:38
Your problem comes probably from the fact that you try to update a value while using `foreach` (namely the `out18` array) which does not make sense. Try using a new vector instead and returning it only. The `.combine` options will cat the result correctly — ClementWalter, Apr 20 '17 at 15:52

score 0 · Answer 1 · edited May 23 '17 at 12:25

I think you're right that an unexpected error is hidden by the tryCatch. For instance, since you're using the lme function, shouldn't you be loading the nlme package on your workers? As in:

r <- foreach(i = 28:78, .packages='nlme', .combine=rbind)  %dopar% {
  #snip
}

There might be other issues, such as objects that have to be explicitly exported, so I suggest that you change your code to not hide unexpected errors, especially while testing/debugging. A simple approach is to log the error and then return the appropriate value:

tryCatch(foo(), error = function(e) {
  print(e)
  NaN
})

In order to see these log messages, you could use the makeCluster outfile="" option so the messages will be displayed in your terminal. See this answer for more information.

I also suggest that you change your test to something like:

if (!identical(lme2, NaN)) ...

This will avoid the warning when lme2 isn't a NaN since identical never returns a value with length > 1.

doParallel seems to corrupts output for unknown reason

1 Answers1