This a follow-up to Parallelize function taking external pointers (XPtr)
I won't reproduce the Cpp code here to make things shorter. The problem was to that once a parameter of a function is evaluated, it is defined in the function's environment and, in the case of an external pointer, is no longer available in a fork cluster.
So while this function worked:
require(parallel)
test1 <- function(a) {
cl <- makeForkCluster(nnodes=2)
r <- parLapply(cl, 1:5, function(i) g(a,i) )
stopCluster(cl)
unlist(r)
}
This function didn't:
test2 <- function(a) {
cl <- makeForkCluster(nnodes=2)
p <- g(a, 0)
r <- parLapply(cl, 1:5, function(i) g(a,i) )
stopCluster(cl)
unlist(r)
}
As pointed out by Ralf Stubner, this came from the fact that the call g(a, 0) forced the evaluation of the promise a. He suggested the following work around (here with two debug prints to understand how it works):
test3 <- function(a) {
cl <- makeForkCluster(nnodes = 2)
print(pryr::promise_info(a))
b <- eval(substitute(a))
p <- g(b, 0)
print(pryr::promise_info(a))
r <- parLapply(cl, 1:5, function(i) g(a,i) )
stopCluster(cl)
unlist(r)
}
This allowed to access to whatever was in a, but a was still an unevaluated promise. But this doesn't work when test3 is called from another function!
test4 <- function(b) test1(b)
test5 <- function(b) test3(b)
While test4 works well (the recursive promise evaluation seems to give a valid pointer), the workaround in test3 no longer works when called from test5.
The debugging prints show that despite the eval(substitute(a)) trick, the promise is evaluated. My understanding is that this tricks forces the evaluation of the promise b of test5 in its environment, thus a becomes evaluated too.
Is there another workaround? (I tried to play with pryr::parent_promise but even the code from the example in the man page gives strange results).
I have other complex problems of this type. A general way to get the content of a promise without evaluating it, or to pass external pointers to other functions, with a parLapply call at the very end, without stumbling constantly on this problem, would be much welcome.
