1

I need to evaluate a Sympy lambda function in parallel using multiprocessing.Pool. It seems Sympy lambda function could not be pickled by default and therefore fails when used in Pool. Is there any solution to it?

A minimal example is as below.

from multiprocessing import Pool
# from pathos.multiprocessing import Pool

x = sympy.symbols('x')
expr = sympy.sympify('x*x')
jacobian_lambda = sympy.lambdify(x, sympy.Matrix([expr]).jacobian([x]))

pool = Pool()

res1 = pool.apply_async(jacobian_lambda, [1])
res2 = pool.apply_async(jacobian_lambda, [2])

print([res1.get(), res2.get()])

I expect to get

[[2], [4]]

which is the evaluation of the derivative of x^2 on 1 and 2.

I got the following error when I try to run the code above.

_pickle.PicklingError: Can't pickle <function <lambda> at 0x7efd90e32d08>: attribute lookup <lambda> on numpy failed

I am aware that Passing sympy lambda to multiprocessing.Pool.map might be a duplicate question, but there does not seem to be a solution without removing lambda function.

Passing sympy lambda to multiprocessing.Pool.map The accepted answer suggests defining a function at the top level, which doesn't seem to solve the problem. Another suggestion to use pathos.multiprocessing does not help in this case, either.

Thanks.

dxli
  • 107
  • 6
  • Possible duplicate of [Python multiprocessing pickling error](https://stackoverflow.com/questions/8804830/python-multiprocessing-pickling-error) – Alexander McFarlane Sep 11 '18 at 13:10
  • You have two options: Try `pathos.multiprocessing` and if that fails write your own caching utility. Make the `sympy` expressions strings and in each function rebuild them as sympy expressions. Then all you send to `multiprocessing` module is the `string` – Alexander McFarlane Sep 11 '18 at 13:12
  • @AlexanderMcFarlane Thanks. I tried `pathos` (as I newly edited), it does not seem to be able to pickle sympy lambda function. I am not totally sure what the second option you propose is. Do you mean I could call `expr.subs()` after builidng an expression into a sympy expression? I might not be able to do so as this slows down the performance a lot. And I am afraid I have to build them into sympy lambda function anyway. – dxli Sep 11 '18 at 13:23
  • yes that's your only real other option – Alexander McFarlane Sep 11 '18 at 13:23
  • I would just pickle the SymPy expression (or re-eval `srepr(expr)` if pickling fails on the SymPy expression, as it sometimes does), and lambdify it separately in each process. lambdification is not the bottleneck, so it shouldn't be an issue to redo it in each process. – asmeurer Sep 11 '18 at 22:46

1 Answers1

0

Switching from multiprocessing to loky seems to do the trick here (much like in the linked question):

from loky import get_reusable_executor
import sympy

x = sympy.symbols('x')
expr = sympy.sympify('x*x')
jacobian_lambda = sympy.lambdify(x, sympy.Matrix([expr]).jacobian([x]))

executor = get_reusable_executor()

list(executor.map(jacobian_lambda, [1,2]))

Out[1]: [array([[2]]), array([[4]])]
mcsoini
  • 6,280
  • 2
  • 15
  • 38