How do I make doSMP play nicely with plyr?

Question

This code works:

library(plyr)
x <- data.frame(V= c("X", "Y", "X", "Y", "Z" ), Z = 1:5)
ddply(x, .(V), function(df) sum(df$Z),.parallel=FALSE)

While this code fails:

library(doSMP)
workers <- startWorkers(2)
registerDoSMP(workers)
x <- data.frame(V= c("X", "Y", "X", "Y", "Z" ), Z = 1:5)
ddply(x, .(V), function(df) sum(df$Z),.parallel=TRUE) 
stopWorkers(workers)

>Error in do.ply(i) : task 3 failed - "subscript out of bounds"
In addition: Warning messages:
1: <anonymous>: ... may be used in an incorrect context: ‘.fun(piece, ...)’

2: <anonymous>: ... may be used in an incorrect context: ‘.fun(piece, ...)’

I am using R 2.1.12, plyr 1.4 and doSMP 1.0-1. Has anyone figured out a way around this?

edit: In response to Andrie, here is a further illustration:

system.time(ddply(x, .(V), function(df) Sys.sleep(1), .parallel=FALSE)) #1
system.time(ddply(x, .(V), function(df) Sys.sleep(1), .parallel=TRUE)) #2
library(doSMP)
workers <- startWorkers(2)
registerDoSMP(workers)
x <- data.frame(V= c("X", "Y", "X", "Y", "Z" ), Z = 1:5)
system.time(ddply(x, .(V), function(df) Sys.sleep(1), .parallel=FALSE)) #3
system.time(ddply(x, .(V), function(df) Sys.sleep(1), .parallel=TRUE)) #4
stopWorkers(workers)

The first three functions work, but they all take about 3 seconds. Function #2 gives a warning that no parallel backend is registered, and thus executes sequentially. Function #4 gives the same error I referenced in my original post.

/edit: curioser and curiouser: On my mac, the following works:

library(plyr)
library(doMC)
registerDoMC()
x <- data.frame(V= c("X", "Y", "X", "Y", "Z" ), Z = 1:5)
ddply(x, .(V), function(df) sum(df$Z),.parallel=TRUE)

But this fails:

library(plyr)
library(doSMP)
workers <- startWorkers(2)
registerDoSMP(workers)
x <- data.frame(V= c("X", "Y", "X", "Y", "Z" ), Z = 1:5)
ddply(x, .(V), function(df) sum(df$Z),.parallel=TRUE) 
stopWorkers(workers)

And this fails too:

library(plyr)
library(snow)
library(doSNOW)
cl <- makeCluster(2, type = "SOCK")
registerDoSNOW(cl)
x <- data.frame(V= c("X", "Y", "X", "Y", "Z" ), Z = 1:5)
ddply(x, .(V), function(df) sum(df$Z),.parallel=TRUE) 
stopCluster(cl)

So I suppose the various parallel back ends for foreach are not interchangeable.

The basic problem is that plyr relies on much information being passed in the enclosure on the `do.ply` function. This data doesn't get passed on by default, and it needs tweaks to the `.export` parameter to work. Still not sure how to do this in general. — hadley, Apr 07 '11 at 13:43
@hadley: it seems like the doMC package is the only one that works seamlessly with plyr. — Zach, Apr 08 '11 at 00:01
@SFun28 I talked with Hadley (the plyr developer) and he confirmed that plyer only works with the doMP backend. Sorry — Zach, Jul 21 '11 at 20:16

score 4 · Accepted Answer · answered Nov 22 '11 at 01:38

4

While the question has been answered well by @hadley, I want to add that I think plyr now works with other foreach parallel back-ends. Here is a link to a blog entry containing an example where plyr is used in conjunction with doSNOW:

answered Nov 22 '11 at 01:38

LeeZamparo

427
4
9

From my reading/experimentation, that link suggests that one needs to define a custom createCluster function not provided by snow. not sure why clusterExport doesn't solve this. Can someone confirm that plyr works with each of the parallel backends now without writing a custom function redifining how they export variables? – cboettig Feb 14 '12 at 00:37
The examples in the link do not work for me. I'm having the same problem. – MeloMCR Feb 24 '15 at 18:24

score 2 · Answer 2 · answered Jan 04 '13 at 23:32

Just to confirm @LeeZamparo's answer, plyr does now seem to work with snow, at least on on Windows 7 with R version 2.15.0. The last chunk of code in the question works, though with cryptic warnings:

library(plyr)
library(snow)
library(doSNOW)
cl <- makeCluster(2, type = "SOCK")
registerDoSNOW(cl)

x <- data.frame(V= c("X", "Y", "X", "Y", "Z" ), Z = 1:5)

library(microbenchmark)
mb <- microbenchmark(

      PP <- ddply(x, .(V), function(df) sum(df$Z),.parallel=TRUE),
      NP <- ddply(x, .(V), function(df) sum(df$Z),.parallel=FALSE) 
                     )

stopCluster(cl)

Cryptic warnings:

> warnings()
Warning messages:
1: <anonymous>: ... may be used in an incorrect context: ‘.fun(piece, ...

It's not quick, I guess that's the overhead...

> mb
Unit: milliseconds
                                                             expr
1 NP <- ddply(x, .(V), function(df) sum(df$Z), .parallel = FALSE)
2 PP <- ddply(x, .(V), function(df) sum(df$Z), .parallel = TRUE)
        min        lq    median        uq       max
1  11.91518  15.74567  20.10944  23.30453  38.09237
2 314.58008 336.81160 348.42421 358.57337 575.11220

Check it gives the expected result

> PP
  V V1
1 X  4
2 Y  6
3 Z  5

Extra details about this session:

> sessionInfo()
R version 2.15.0 (2012-03-30)
Platform: i386-pc-mingw32/i386 (32-bit)

locale:
[1] LC_COLLATE=English_Australia.1252  LC_CTYPE=English_Australia.1252   
[3] LC_MONETARY=English_Australia.1252 LC_NUMERIC=C                      
[5] LC_TIME=English_Australia.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] microbenchmark_1.1-3 doSNOW_1.0.6         iterators_1.0.6     
[4] foreach_1.4.0        plyr_1.7.1           snow_0.3-10          

loaded via a namespace (and not attached):
[1] codetools_0.2-8 compiler_2.15.0 tools_2.15.0

score 1 · Answer 3 · edited May 23 '17 at 12:24

1

It turns out plyr only works with doMC, but the developer is working on it.

edited May 23 '17 at 12:24

Community

1
1

answered Apr 13 '11 at 17:33

Zach

29,791
35
142
201

How do I make doSMP play nicely with plyr?

3 Answers3

Linked