How to make a object available to node in the snow package for R parallel computing

Question

This is something I find difficult to understand:

cl = makeCluster(rep("localhost", 8), "SOCK")

# This will not work, error: dat not found in the nodes
pmult = function(cl, a, x)
{
    mult = function(s) s*x
    parLapply(cl, a, mult)
}
scalars = 1:4
dat = rnorm(4)
pmult(cl, scalars, dat)

# This will work
pmult = function(cl, a, x)
{
    x
    mult = function(s) s*x
    parLapply(cl, a, mult)
}
scalars = 1:4
dat = rnorm(4)
pmult(cl, scalars, dat)

# This will work
pmult = function(cl, a, x)
{
    mult = function(s, x) s*x
    parLapply(cl, a, mult, x)
}
scalars = 1:4
dat = rnorm(4)
pmult(cl, scalars, dat)

The first function doesn't work because of lazy evaluation of arguments. But what is lazy evaluation? When mult() is executed, does it not require x to be evaluated? The second one works because it forces x to be evaluated. Now the most strange thing happens in the third function, nothing is done but make mult() receive x as an extra argument, and suddenly everything works!

Another thing is, what should I do if I don't want to define all the variables and functions inside the function calling parLapply()? The following definitely will not work:

pmult = function(cl)
{
    source("a_x_mult.r")
    parLapply(cl, a, mult, x)
}
scalars = 1:4
dat = rnorm(4)
pmult(cl, scalars, dat)

I can pass all these variables and functions as arguments:

f1 = function(i)
{
    return(rnorm(i))
}

f2 = function(y)
{
    return(f1(y)^2)
}

f3 = function(v)
{
    return(v- floor(v) + 100)
}

test = function(cl, f1, f2, f3)
{
    x = f2(15)
    parLapply(cl, x, f3)
}

test(cl, f1, f2, f3)

Or I can use clusterExport(), but it'll be cumbersome when there are lots of objects to be exported. Is there a better way?

See `?force` for some explanation. – Joshua Ulrich May 31 '13 at 16:27 — Joshua Ulrich, May 31 '13 at 16:27

Steve Weston · Accepted Answer · 2013-05-31T20:51:03.013

To understand this, you have to realize that there is an environment associated with every function, and what that environment is depends on how the function was created. A function that is simply created in a script is associated with the global environment, but a function that is created by another function is associated with the local environment of the creating function. In your example, pmult creates mult, so the environment associated with mult contains the formal arguments cl, a, and x.

The problem with the first case is that parLapply doesn't know anything about x: it is just an unevaluated formal argument that is serialized as part of the environment of mult by parLapply. Since x isn't evaluated when mult is serialized and sent to the cluster workers, it causes an error when the workers execute mult, since dat isn't available in that context. In other words, by the time mult evaluates x, it's too late.

The second case works because x is evaluated before mult is serialized, so the actual value of x is serialized along with the environment of mult. It does what you would expect if you knew about closures but not lazy argument evaluation.

The third case works because you're having parLapply handle x for you. There's no trickery going on at all.

I should warn you that in all of these cases, a is being evaluated (by parLapply) and serialized along with the environment of mult. parLapply is also splitting a into chunks and sending those chunks to each worker, so the copy of a in the environment of mult is completely unnecessary. It doesn't cause an error, but it could hurt performance, since mult is sent to the workers in every task object. Fortunately, this is much less of a problem with parLapply, since there is only one task per worker. It would be a much worse problem with clusterApply or clusterApplyLB where the number of tasks is equal to the length of a.

I talk about a number of issues relating to functions and environments in the "snow" chapter of my book. There are some subtle issues involved, and it's easy to get burned, sometimes without realizing that it happened.

As for your second question, there are various strategies for exporting functions to the workers, but some people do use source to define functions on the workers rather than using clusterExport. Keep in mind that source has a local argument that controls where the parsed expressions are evaluated, and you may need to specify the absolute path to the script. Finally, if you're using remote cluster workers, you may need to scp the script to the workers if you don't have a distributed file system.

Here is a simple method of exporting all of the functions in your global environment to the cluster workers:

ex <- Filter(function(x) is.function(get(x, .GlobalEnv)), ls(.GlobalEnv))
clusterExport(cl, ex)

It's an honor to have your answer here, sir! I am reading _Parallel R_ right now. — qed, May 31 '13 at 19:28
@CravingSpirit I hope you find this answer and the book useful. — Steve Weston, May 31 '13 at 20:52
Thanks. On page 23 of the book, in the footnote, you wrote: "Actually, if you specify the worker function by name, rather than by providing the definition of the function, most of the parallel execution functions will use that name to look up that function in the ... thus avoiding serialization". I still don't quite understand the difference between "specify by name" and "provide definition", could you please explain a little? — qed, Jun 03 '13 at 13:29
Also in this snippet of code: `a = 1:4; x = rnorm(4); clusterExport(cl, x); mult = function(s) s*x; parLapply(cl, a, mult) `, why do we need export `x` but not `mult` when they are both in top level? — qed, Jun 03 '13 at 13:31
@CravingSpirit If a function is defined in a package that has been loaded on the workers, you can execute it using it's name rather than the function object itself, as in: `parLapply(cl, 1:10, 'sqrt')`. In that case, you never actually serialized it. — Steve Weston, Jun 03 '13 at 17:42
@CravingSpirit When you execute `parLapply(cl, a, mult)`, `parLapply` will serialize `mult` and use it for that operation, so it doesn't need to be exported. But `parLapply` isn't really aware of `x` in this case, so you have to export it yourself. However, if you did export `mult`, you could then execute it repeatedly using `parLapply(cl, a, 'mult')`, thus avoiding sending it to the cluster workers repeatedly since you're specifying the function by name. — Steve Weston, Jun 03 '13 at 17:50

How to make a object available to node in the snow package for R parallel computing

1 Answers1