weird behavior of multiprocessing scipy optimization inside of a function

Question

Here is a simple code that runs well. Even if the function minimize wraps the scipy.optimize.minimize it does not complain about pickling

import numpy as np
from scipy import optimize
from multiprocessing import Pool

def square(x):
    return np.sum(x**2+ 2*x)

def minimize(args):
    f,x = args
    res = optimize.minimize(f, x, method = 'L-BFGS-B')
    return res.x

x = np.random.rand(8,10)

args = [(square,x[i]) for i in range(8)]
p = Pool(8)
p.map(minimize,args)

However, if try the following it fails with the pickling error

PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

def run():
    def square(x):
        return np.sum(x**2+ 2*x)

    def minimize(args):
        f,x = args
        res = optimize.minimize(f, x, method = 'L-BFGS-B')
        return res.x

    x = np.random.rand(8,10)

    args = [(square,x[i]) for i in range(8)]
    p = Pool(8)
    p.map(minimize,args)

run()

I want to make a module to use scipy minimize in parallel with many population of initial guesses. However, as shown in the example, when I make it a module, it fails.

The `pickle` module can only pick top-level functions. In your second example, `square` is not a top-level function, so it fails to pickle it. — dano, Sep 29 '16 at 18:17
See [this question](http://stackoverflow.com/questions/12019961/python-pickling-nested-functions). — dano, Sep 29 '16 at 18:17
Thank you @dano. But the same error occurs if I take square out of the box. I mean just before the definition of run(). Doesn't it make it top-level? — Kilean Hwang, Sep 29 '16 at 18:45
You'll have to to the same with `minimize`, since you're passing that to `map`, too. Sorry, should have noticed that to begin with. — dano, Sep 29 '16 at 19:40

score 2 · Accepted Answer · edited May 23 '17 at 10:34

The problem is that Python cannot pickle nested functions, and in your second example, you're trying to pass the nested minimize and square functions to your child process, which requires pickling.

If there's no reason that you must nest those two functions, moving them to the top-level of the module will fix the issue. You can also see this question for techniques to pickle nested functions.

weird behavior of multiprocessing scipy optimization inside of a function

1 Answers1

Linked