scipy.optimize.differential_evolution won't work in parallel with workers

Question

I'm trying to minimise my __optimisation_function in parallel setting workers=-1 when calling differential_evolution with scipy=1.10.1 and Python 3.9.7 (macOS Ventura 13.4), but I'm getting the error below.

Any idea how to make this work in parallel?

df_1 and df_2 are pandas DataFrames.

Removing the workers=-1 from differential_evolution make it work just fine.

from scipy.optimize import differential_evolution

class Optimisation():

    def __optimisation_function(self, x, *args) -> float:

        a, b, c, d, e = x

        df_1 = args[0]
        df_2 = args[1]

        return something

    def run_optimisation(self):

        optimisation_result = differential_evolution(
            func=self.__optimisation_function,
            bounds=bounds,
            constraints=constraints,
            args=(df_1, df_2),
            workers=-1
        )

Error

.../python3.9/site-packages/scipy/optimize/_differentialevolution.py:382: UserWarning: differential_evolution: the 'workers' keyword has overridden updating='immediate' to updating='deferred'
  with DifferentialEvolutionSolver(func, bounds, args=args,
Process SpawnPoolWorker-2:
Traceback (most recent call last):
  File ".../python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File ".../python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File ".../python3.9/multiprocessing/pool.py", line 114, in worker
    task = get()
  File ".../python3.9/multiprocessing/queues.py", line 368, in get
    return _ForkingPickler.loads(res)
AttributeError: 'Optimisation' object has no attribute '__optimisation_function'
Process SpawnPoolWorker-1:
Traceback (most recent call last):
  File ".../python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File ".../python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File ".../python3.9/multiprocessing/pool.py", line 114, in worker
    task = get()
  File ".../python3.9/multiprocessing/queues.py", line 368, in get
    return _ForkingPickler.loads(res)
AttributeError: 'Optimisation' object has no attribute '__optimisation_function'

How are you running this? The documentation does mention that the `func` must be pickleable, because that's how it passes the function to the subprocesses. I don't know whether a closure like that is pickleable. — Tim Roberts, Jul 08 '23 at 01:12
Please give a MWE when seeking help, otherwise it's difficult to help. The test suite for `optimize.differential_evolution` covers parallelisation, and this functionality is used frequently. It's likely to be a problem with the way you set your system up, rather than with `differential_evolution` itself. — Andrew Nelson, Jul 09 '23 at 07:31
In this case one of the issues here is probably the double underscore doing some name-mangling. — Andrew Nelson, Jul 09 '23 at 07:46

score 0 · Accepted Answer · answered Jul 09 '23 at 10:05

Modifying my function name from __optimisation_function to optimisation_function solved my issue.

Calling

optimisation_result = differential_evolution(
    func=Optimisation._Optimisation__optimisation_function,
    bounds=bounds,
    constraints=constraints,
    args=(df_1, df_2),
    workers=-1
)

with func=Optimisation._Optimisation__optimisation_function instead of func=self.__optimisation_function

Returned

Exception has occurred: PicklingError
Can't pickle <function Optimisation.__optimisation_function at 0x7fa17107e040>: attribute lookup Optimisation.__optimisation_function on source.optimiser failed

Hence, I had to change the method name as __optimisation_function did not seem to be pickleable.

It turned out to be an issue with how pickle looks for __name__ to serialize the method.

This issue is explained in further detail here.

scipy.optimize.differential_evolution won't work in parallel with workers

1 Answers1