python scipy differential evolution optimization fails with workers not 1

Question

I am trying to use the workers parameter of the scipy differential evolution algo.

When I put it as 1, my script runs with no issues. If I put something different, it fails with the following traceback :

Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.6/site-packages/scipy/_lib/_util.py", line 419, in __call__
    return self._mapfunc(func, iterable)
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 266, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 644, in get
    raise self._value
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 424, in _handle_tasks
    put(task)
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/usr/lib/python3.6/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
TypeError: cannot serialize '_io.TextIOWrapper' object

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.6/site-packages/scipy/optimize/_differentialevolution.py", line 878, in _calculate_population_energies
    parameters_pop[0:nfevs]))
  File "/home/ubuntu/.local/lib/python3.6/site-packages/scipy/_lib/_util.py", line 422, in __call__
    raise TypeError("The map-like callable must be of the"
TypeError: The map-like callable must be of the form f(func, iterable)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main_parallel.py", line 323, in <module>
    optimizer(args.asset_class, udl, tenor, args.criteria, params_decay, params, couch_server, args.num_workers)
  File "main_parallel.py", line 269, in optimizer
    maxiter=5, workers=num_workers, mutation=(0.5, 1.5), recombination=0.8)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/scipy/optimize/_differentialevolution.py", line 306, in differential_evolution
    ret = solver.solve()
  File "/home/ubuntu/.local/lib/python3.6/site-packages/scipy/optimize/_differentialevolution.py", line 745, in solve
    self.population[self.feasible]))
  File "/home/ubuntu/.local/lib/python3.6/site-packages/scipy/optimize/_differentialevolution.py", line 883, in _calculate_population_energies
    raise RuntimeError("The map-like callable must be of the"
RuntimeError: The map-like callable must be of the form f(func, iterable), returning a sequence of numbers the same length as 'iterable'

Could someone please help in explaining how to use this parameter (specific objective function ? other constraints ?) ?

Also an example of a simple python differential evolution optimization using multiple cores would be appreciated.

William Miller · Accepted Answer · 2020-01-15T05:17:29.847

2

The critical piece of the differential evolution documentation regarding the usage of workers is the caveat that when worker != 1, it

[...] Requires that func be pickleable

There are several hints in the errors raised which indicate the func is not pickleable, namely

    self._mapfunc(func, iterable)
    ...
    self._send_bytes(_ForkingPickler.dumps(obj))
    ...
TypeError: cannot serialize '_io.TextIOWrapper' object

Clearly, there was an attempt at pickling func made which failed due, presumably, to func not being pickleable. It looks like workers is attempted to be interpreted as a map-like or callable, as the other signature in the documentation indicates. Predictably this fails as well since workers is in fact an int.

The documentation contains a complete example where workers != 1 which works correctly,

from scipy.optimize import differential_evolution, rosen
bounds = [(0,2), (0, 2), (0, 2), (0, 2), (0, 2)]
result = differential_evolution(rosen, bounds, updating='deferred',
                                workers=-1)
result.x, result.fun

If you refactor the func being used in scipy.optimize.differential_evolution to be serializable via pickling you should be able to use workers != 1 successfully.

edited Jan 15 '20 at 05:17

answered Jan 15 '20 at 03:52

William Miller

9,839
3
25
46

Thanks for your help. How do you know that the culprit is args.asset_class ? It's just a string so I would assume its handling is built-in. – Chapo Jan 15 '20 at 05:13
@Chapo That is a mistake, I misread the error statement. I would need to see how you call `scipy.optimize.differential_evolution` to tell you what needs to be refactored for pickling – William Miller Jan 15 '20 at 05:16
`result_diff_evo = differential_evolution(func=optimizer_function, bounds=bounds, args=( init_params_names, df, criteria, params_decay[udl][tenor]["first_date"], params_decay[udl][tenor]["busdays"], params_decay[udl][tenor]["coeff"], params_decay[udl][tenor]["spot_start"], params_decay[udl][tenor]["iv_start"], params["verbose"], folder, tested_params,), init=pop_denorm, maxiter=5, workers=num_workers, mutation=(0.5, 1.5), recombination=0.8, print=True)` is the actual line – Chapo Jan 15 '20 at 05:18
@Chapo The solution to your problem is to refactor `optimizer_function` to be serializable via pickling – William Miller Jan 15 '20 at 05:19
all of the params are fairly simple except for `tested_params` which is obtained through `tested_params = open(f"tested_params.csv", "w")`. Could it be the issue ? – Chapo Jan 15 '20 at 05:19
ok fair enough. Will have to check if i can pickle an open file – Chapo Jan 15 '20 at 05:20
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/205981/discussion-between-chapo-and-william-miller). – Chapo Jan 15 '20 at 05:21
@Chapo Good to hear it, glad I could help – William Miller Jan 28 '20 at 07:31

python scipy differential evolution optimization fails with workers not 1

1 Answers1