python multiprocessing : AttributeError: Can't pickle local object

Question

I have a method inside a class to return a func which parameters may change.

The Interface function accept two parameters, f and its args.I want to use mp.pool to accelerate it.However, it returns an error.

from multiprocessing import Pool
# from multiprocess import Pool
# from pathos.multiprocessing import ProcessingPool as Pool
import pickle
import dill


class Temp:
    def __init__(self, a):
        self.a = a

    def test(self):
        def test1(x):
            return self.a + x

        return test1


def InterfaceFunc(f, x):
    mypool = Pool(4)
    return list(mypool.map(f, x))


if __name__ == "__main__":
    t1 = Temp(1).test()
    x = [1, 2, 3, 1, 2]

    res1 = list(map(t1, x))
    print(res1)

    res2 = InterfaceFunc(t1, x)

it raise the same error:

AttributeError: Can't pickle local object 'Temp.test.<locals>.test1'

I have tried 3 method:

What can multiprocessing and dill do together?

Replace pickle in Python multiprocessing lib

Python Multiprocessing Pool Map: AttributeError: Can't pickle local object

Method 1, 2 :

 from multiprocess import Pool
 from pathos.multiprocessing import ProcessingPool as Pool

It raise error:

 File "E:\Users\ll\Anaconda3\lib\site-packages\dill\_dill.py", line 577, in _load_type
    return _reverse_typemap[name]
KeyError: 'ClassType'

Method3 needs to change the code , however I cant simply move the func out of the class because I need f to be a parameter for the Interface.

Do you have any suggestions? I'm an inexperienced newcomer.

Your example doesn't run which makes it harder to experiment with solutions. — tdelaney, Jun 04 '20 at 03:20
@tdelaney sorry, I made a mistake before. I have changed the code and it raises an exception. — abcyunice, Jun 04 '20 at 03:33
I don't see how the linked question answers this one. There, OP could move the function out to module level but that's not the case here because the call also has state that needs to be managed. — tdelaney, Jun 04 '20 at 05:58

tdelaney · Accepted Answer · 2020-06-04T05:46:15.953

Python can't pickle the closure, but all you really need is something that you can call that retains state. The __call__ method makes a class instance callable, so use that

from multiprocessing import Pool

class TempTest1:

    def __init__(self, a):
        self.a = a

    def __call__(self, x):
        return self.a + x

class Temp:
    def __init__(self, a):
        self.a = a

    def test(self):
        return TempTest1(self.a)

def InterfaceFunc(f, x):
    mypool = Pool(4)
    return list(mypool.map(f, x))

if __name__ == "__main__":
    t1 = Temp(1).test()
    x = [1, 2, 3, 1, 2]

    res1 = list(map(t1, x))
    print(res1)

    res2 = InterfaceFunc(t1, x)
    print(res2)

Thanks for the suggestion. Would you share some document of "Python can't pickle the closure,"? I couldn't find related information at "https://www.python.org/". — Cloud Cho, Jul 15 '21 at 17:56

score 1 · Answer 2 · answered Jan 06 '23 at 09:49

A bit late to the party, but passing the class function (the one you want) into a wrapper function did the trick for me:

from multiprocessing import Pool

def myWorkFunc(load):
    return load["fu"](load["i"])

class Test():
    def __init__(self):
        self.a = 1

    def add_one(self, x):
        print('adding', x)
        return x + 1

    def do_something(self):
        loads = [{"i": i, "fu": self.add_one} for i in range(4)]
        with Pool(4) as mypool:
            return mypool.map(func=myWorkFunc, iterable=loads)

if __name__ == '__main__':
    T = Test()
    print(T.do_something())

score 0 · Answer 3 · answered Feb 21 '23 at 01:02

I tested multiple ways to handle this error and I think the easiest one would be to get to the file that is giving you this error and substitute 1 with 2.

import pickle
import dill as pickle

dill library extends the features of the pickle and will handle this error for you.

Good Luck!

python multiprocessing : AttributeError: Can't pickle local object

3 Answers3

Linked