Python Multiprocessing Can't pickle local object

Question

i have read a little about multiprocessing and pickling problems, I have also read that there are some solutions but I don't really know how can they help to mine situation.

I am building Test Runner where I use Multiprocessing to call modified Test Class methods. Modified by metaclass so I can have setUp and tearDown methods before and after each run test.

Here is my Parent Metaclass:

class MetaTestCase(type):

def __new__(cls, name: str, bases: Tuple, attrs: dict):
    
    def replaced_func(fn):
        def new_test(*args, **kwargs):
            args[0].before()
            result = fn(*args, **kwargs)
            args[0].after()
            return result

        return new_test

    # If method is found and its name starts with test, replace it
    for i in attrs:
        if callable(attrs[i]) and attrs[i].__name__.startswith('test'):
            attrs[i] = replaced_func(attrs[i])

    return (super(MetaTestCase, cls).__new__(cls, name, bases, attrs))

I am using this Sub Class to inherit MetaClass:

class TestCase(metaclass=MetaTestCase):

def before(self) -> None:
    """Overridable, execute before test part."""
    pass

def after(self) -> None:
    """Overridable, execute after test part."""
    pass

And then I use this in my TestSuite Class:

class TestApi(TestCase):

def before(self):
    print('before')

def after(self):
    print('after')

def test_api_one(self):
    print('test')

Sadly when I try to execute that test with multiprocessing.Process it fails on

AttributeError: Can't pickle local object 'MetaTestCase.__new__.<locals>.replaced_func.<locals>.new_test'

Here is how I create and execute Process:

module = importlib.import_module('tests.api.test_api')  # Finding and importing module
object = getattr(module, 'TestApi')  # Getting Class from module

process = Process(target=getattr(object, 'test_api_one')) # Calling class method
process.start()
process.join()

I tried to use pathos.helpers.mp.Process, it passes pickling phase I guess but has some problems with Tuple that I don't understand:

Process Process-1:
Traceback (most recent call last):
    result = fn(*args, **kwargs)
IndexError: tuple index out of range

Is there any simple solution for that so I can pickle that object and run test sucessfully along with my modified test class?

In fact, the pickling of local functions is something that `multiprocess` can do (it is the same as the `multiprocess` provided by `pathos`) and that the standard `multiprocessing` cannot. This `IndexError` looks to me as if you were not passing the right `args` to the `fn` function. If your `fn` is something of the type `self.my_test_method`, and `self` is `args[0]`, it could be that you would need to be passing `*args[1:]` to `fn` instead of `*args`. — molinav, Sep 01 '22 at 14:05
Do you get the same error if trying to call the `test_api_one` function directly in your main script, without any use of multiprocessing? — molinav, Sep 01 '22 at 14:08
@molinav If I call test directly by using: TS = TestSuite(); TS.test() I get expected behaviour of "before Test after" Tried to pass *args[1:] but nothing changed cause it seems like there is nothing in that Tuple at all :/ — Alraku, Sep 01 '22 at 14:22

Charchit Agarwal · Accepted Answer · 2022-09-01T20:47:31.043

As for your original question of why you are getting the pickling error, this answer summarizes the problem and offers solutions (similar to those already provided here).

Now as to why you are receiving the IndexError, this is because you are not passing an instance of the class to the function (the self argument). A quick fix would be to do this (also, please don't use object as a variable name):

module = importlib.import_module('tests.api.test_api')  # Finding and importing module
obj = getattr(module, 'TestApi')
test_api = obj()  # Instantiate!

# Pass the instance explicitly! Alternatively, you can also do target=test_api.test_api_one
process = Process(target=getattr(obj, 'test_api_one'), args=(test_api, ))  
process.start()
process.join()

Ofcourse, you can also opt to make the methods of the class as class methods, and pass the target function as obj.method_name.

Also, as a quick sidenote, the usage of a metaclass for the use case shown in the example seems like an overkill. Are you sure you can't do what you want with class decorators instead (which might also be compatible with the standard library's multiprocessing)?

Hello @Charchit, first of all thank you for your answer as it helped me - marked as a accepted answer. Secondly - I tried moving out the nested function out of the class as provided in your link, but I had no idea how to do that and failed with the same error (index error). Now I understand what I was lacking... Answering to the sidenote - it may look like an overkill, but the goal of project that I am developing is to create test runner with built in functions like parallel execution and GUI. I didn't want to use decorators to mark every test method as it would be too much. — Alraku, Sep 02 '22 at 06:46

score -1 · Answer 2 · answered Sep 01 '22 at 13:55

https://docs.python.org/3/library/pickle.html#what-can-be-pickled-and-unpickled

"The following types can be pickled... functions (built-in and user-defined) accessible from the top level of a module (using def, not lambda);"

It sounds like you cannot pickle locally defined functions. This makes sense based on other pickle behavior I've seen. Essentially it's just pickling instructions to the python interpreter for how it can find the function definition. That usually means its a module name and function name (for example) so the multiprocessing Process can import the correct function.

There's no way for another process to import your replaced_func function because it's only locally defined.

You could try defining it outside of the metaclass, which would make it importable by other processes.

Tried without pathos.Process and moved replaced_func out of the class - the behaviour is like at the end of the Original Post which is "IndexError: tuple index out of range". I am wondering if that solved the issue with pickling, so my next error is about not passing arguments properly? — Alraku, Sep 01 '22 at 14:24

Python Multiprocessing Can't pickle local object

2 Answers2