Passing sympy lambda to multiprocessing.Pool.map

Question

I want to execute a sympy lambda function in parallel. I don't know:

why it works in parallel although it is a lambda function
why it stops working when I try executing without the pool
why it works if I uncomment the first return in lambdify

And apparently the markdown preprocessor needs a line of text above the code so this is the code:

from multiprocessing import Pool

import sympy
from sympy.abc import x

def f(m):
    return m.lambdify()(1)

class Mult():
    def lambdify(self):
        # return sympy.lambdify(x, 2*x, 'numpy')
        self._lambdify = sympy.lambdify(x, 2 * x, 'numpy')
        return self._lambdify

if __name__ == '__main__':
    with Pool() as pool:
        m = Mult()
        print(pool.map(f, [m]))
        print(pool.map(f, [m]))
        print(f(m))
        print(pool.map(f, [m]))

It prints:

[2]
[2]
2
PicklingError: Can't pickle <function <lambda> at 0x000000000DF0D048>: attribute lookup <lambda> on numpy failed

(I cut the traceback)

If I uncomment, it works normally:

[2]
[2]
2
[2]

I tested only on Windows and it works exactly the same with 'numexpr' instead of 'numpy'.

You cannot call methods on instances of _live_ objects across processes using `multiprocessing.Pool.map()` unless you go out of your way to explain the targeted subprocess how to reconstruct your _live_ object on its side. For an example, check [this answer](https://stackoverflow.com/a/44186168/7553525). — zwer, Jul 26 '17 at 13:29
@zwer Of course, but it does not explain the points 2 and 3. — Labo, Jul 26 '17 at 15:02

Thomas Moreau · Accepted Answer · 2017-07-27T10:56:59.677

2

The object Mult has no fields when it is created. It can thus be pickled with the stock pickle library. Then, when you call lambdify, you add a _lambdify attribute to the object containing a lambda expression, which cannot be pickled. This causes a failure in the map function

This explains why before calling lambdify you can pickle the object and use Pool.map and why it fails after the call. When you uncomment the line in lambdify, you do not add the attribute to the class, and the Mult object can still be pickled after calling lambdify.

edited Jul 27 '17 at 10:56

answered Jul 27 '17 at 08:30

Thomas Moreau

4,377
1
20
32

It sounds like it explains everything. Do you have a reference for "you add a _lambdify attribute to the object containing a lambda expression, which cannot be pickled"? – Labo Jul 27 '17 at 09:35
[This reference](https://docs.python.org/3/library/pickle.html#id2) explain why lambda expression cannot be pickled. Note that apparently, [`dill`](https://github.com/uqfoundation/dill) can serialize the result from `sympy.lambdify`. – Thomas Moreau Jul 27 '17 at 09:55
My bad, `dill` also fails to pickle `sympy` objects. – Thomas Moreau Jul 27 '17 at 10:57
Oh excuse me, I didn't understand what you said the first time. You are totally right. The thing that changes is that I try to pickle an object that contains a lambda. – Labo Jul 27 '17 at 12:57

score 1 · Answer 2 · answered Nov 11 '19 at 15:26

Though I have not fully explored this yet, I just want to put on record that the same example works just fine when using loky instead of multiprocessing:

from loky import get_reusable_executor

import sympy
from sympy.abc import x

def f(m):
    return m.lambdify()(1)

class Mult():
    def lambdify(self):
#        return sympy.lambdify(x, 2*x, 'numpy')
        self._lambdify = sympy.lambdify(x, 2 * x, 'numpy')
        return self._lambdify


executor = get_reusable_executor()

m = Mult()
print('pool.map(f, [m])', list(executor.map(f, [m])))
print('pool.map(f, [m])', list(executor.map(f, [m])))
print('f(m)', f(m))
print('pool.map(f, [m])', list(executor.map(f, [m])))

with output

pool.map(f, [m]) [2]
pool.map(f, [m]) [2]
f(m) 2
pool.map(f, [m]) [2]

Thank you! Indeed, loky is amazing. The interface of [joblib](https://joblib.readthedocs.io/en/latest/) is quite useful too. — Labo, Nov 11 '19 at 20:07

Passing sympy lambda to multiprocessing.Pool.map

2 Answers2

Linked