DEoptim error: objective function result has different length than parameter matrix due to foreachArgs specification

Question

I have a very odd DEoptim error that I have "fixed", but do not understand.

I do not have issues when I use DEoptim's parallel capabilities with the parallel package (i.e., pType=1). However when I use foreach instead (which I must use on the grid computing setup that is available to me), I have issues. Below is an MRE of a much simplified version of the issue that I had. pType=1 works, pType=2 when foreachArgs is specified returns an error:

objective function result has different length than parameter matrix

When I do not specify foreachArgs the issue goes away. Does anyone have thoughts about the root cause of this issue?

library(zoo)
library(parallel)
library(doParallel)
library(DEoptim)


myfunc1 <- function(params){
    s <- myfunc2(params,ncal,n_left_cens,astats, X_ret, disc_length, X_acq, POP_0, POP_ann_growth)
    loss_func(s)
    }

myfunc2 = function(params,ncal,n_left_cens,astats, X_ret, disc_length, X_acq, POP_0, POP_ann_growth){
    sum(params) + ncal + n_left_cens + astats + X_ret + disc_length + X_acq + POP_0 + POP_ann_growth 
    }

loss_func = function(s){
    s
    }   


# General setup
ncal = 1
n_left_cens = 1
astats= 1
disc_length = 1
POP_0 = 1
POP_ann_growth = 1
X_acq = 1
X_ret = 1
params = c(1,1)
W = 1

paral = TRUE
itermax=100
ncores = detectCores()
cltype <- ifelse(.Platform$OS.type != "windows", "FORK", "PSOCK")
trace=TRUE

# bounds for search for DEoptim
lower = rep(-1,length(params))
upper = lower*-1




# parallel: works               
pType = 1
parVar = c("myfunc1","myfunc2","loss_func","W","ncal","n_left_cens","astats","X_ret","disc_length",
                        "X_acq","POP_0","POP_ann_growth")               
foreachArguments <- list("myfunc1","myfunc2","loss_func","ncal","n_left_cens","astats","X_ret","disc_length",
                        "X_acq","POP_0","POP_ann_growth")
clusters <- makeCluster(ncores, type = cltype)
registerDoParallel(clusters)
clusterExport(cl=clusters, varlist=foreachArguments, envir=environment())
results <- DEoptim(fn=myfunc1,lower=lower,upper=upper,
                DEoptim.control(itermax=itermax,trace=trace,parallelType=pType,
                parVar=parVar))
showConnections(all = TRUE)
closeAllConnections()





# foreach with foreachArgs specified: doesn't work
pType = 2               
clusters <- makeCluster(ncores, type = cltype)
registerDoParallel(clusters)
clusterExport(cl=clusters, varlist=foreachArguments, envir=environment())

results <- DEoptim(fn=myfunc1,lower=lower,upper=upper,
                DEoptim.control(itermax=itermax,trace=trace,parallelType=pType,
                foreachArgs=foreachArguments))
showConnections(all = TRUE)
closeAllConnections()





# foreach with foreachArgs unspecified: works
pType = 2               
foreachArguments <- list("myfunc1","myfunc2","loss_func","ncal","n_left_cens","astats","X_ret","disc_length",
                        "X_acq","POP_0","POP_ann_growth")
clusters <- makeCluster(ncores, type = cltype)
registerDoParallel(clusters)
clusterExport(cl=clusters, varlist=foreachArguments, envir=environment())

results <- DEoptim(fn=myfunc1,lower=lower,upper=upper,
                DEoptim.control(itermax=itermax,trace=trace,parallelType=pType))
showConnections(all = TRUE)
closeAllConnections()

When I run your code, your first example fails with, "Error in clusterExport(...) object 'foreachArguments' not found"; and your second example fails with, "Error in parallelArgs$.packages : object of type 'symbol' is not subsettable". So you need to provide a reproducible example. — Joshua Ulrich, Jul 05 '16 at 03:45
My apologies Josh - 'foreachArguments' was defined below when it should have been defined above. Edit made. On a fresh R session, the results are as they had been described -- works, doesn't work giving that error message, then works... — user052084, Jul 05 '16 at 12:48

Joshua Ulrich · Accepted Answer · 2016-07-06T04:05:35.303

From ?DEoptim.control:

foreachArgs: A list of named arguments for the ‘foreach’ function from the package ‘foreach’. The arguments ‘i’, ‘.combine’ and ‘.export’ are not possible to set here; they are set internally.

Which you seem to be conflating with the behavior of parVar:

parVar: Used if ‘parallelType=1’; a list of variable names (as strings) that need to exist in the environment for use by the objective function or are used as arguments by the objective function.

You need to specify the arguments passed to foreach as name = value pairs. For example:

foreachArguments <- list(.export = c("myfunc1", "myfunc2", "loss_func", "ncal",
 "n_left_cens", "astats", "X_ret", "disc_length", "X_acq","POP_0","POP_ann_growth")

I'm not sure what's causing that specific error, but the fix is "don't do that." ;)

Here's an example of how you'd actually use the foreachArgs argument. Note that I'm setting the .verbose argument to make foreach print diagnostics:

library(doParallel)
library(DEoptim)
clusters <- makeCluster(detectCores())
registerDoParallel(clusters)
obj_func <- function(params) { sum(params) }
results <- DEoptim(fn=obj_func, lower=c(-1, -1), upper=c(1, 1),
  DEoptim.control(parallelType=2, foreachArgs=list(.verbose=TRUE)))
stopCluster(clusters)

Thank you Josh! While foreach didn't export variables for me, your specification of foreachArguments did the trick when using foreachArgs. It provided the same results as what I obtain by exporting manually and not using foreachArgs. — user052084, Jul 06 '16 at 01:53
@user052084: I'm not certain `foreach` automatically exports variables, so I removed that comment from my answer. Sorry for any confusion. — Joshua Ulrich, Jul 06 '16 at 04:06

DEoptim error: objective function result has different length than parameter matrix due to foreachArgs specification

1 Answers1