Dynamically creating serializable classes in Python

Question

I am trying to update a class that is supposed to be used as a custom type for the multiprocessing.manager and imitate a basic dictionary. All works well on Linux, but things fail on Windows and I understood that the problem lies in a possibly suboptimal creation mechanism that it uses that involves a closure. With forking, Linux gets around serializing something that pickle cannot cope with, while this does not happen on Windows. I am using Python 3.6 and feel like it is better to improve the class rather than force a new package dependency that has more robust serialization than pickle.

An example that I think demonstrates this is presented below. It involves a class that is meant to act like a dict, but have an additional method and a class attribute. These are bound in a factory method that the code calls and passes to multiprocessing.manager.register. I get AttributeError: Can't pickle local object 'foo_factory.<locals>.Foo' as a result here.

import abc
import pickle


class FooTemplate(abc.ABC, dict):

    bar = None

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.foo = 'foo'

    @abc.abstractmethod
    def special_method(self, arg1, arg2):
        pass


def foo_factory(dynamic_special_method):

    class Foo(FooTemplate):
        bar = 'bar'

        def special_method(self, arg1, arg2):
            print(self.foo, ' ', self.bar, ' ', dynamic_special_method(arg1, arg2))

    return Foo


def method_to_pass(a1, a2):
    return a1 + a2


if __name__ == '__main__':
    foo = foo_factory(method_to_pass)()
    pickle.dumps(foo)

I attempted to fix the problem by creating a class dynamically, but this throws a new error that I am not sure I understand and it makes things look even worse with all honesty. Using the main part from above with the code below produces error _pickle.PicklingError: Can't pickle <class '__main__.Foo'>: attribute lookup Foo on __main__ failed.

class FooTemplate(dict):

    bar = None
    method_map = None

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.foo = 'foo'

    def special_method(self, arg1, arg2):
        print(self.foo, ' ', self.bar, ' ', self.method_map[self.bar](arg1, arg2))


def foo_factory(dynamic_special_method):
    return type('Foo', (FooTemplate,), {'bar': 'bar', 'method_map': {'bar': dynamic_special_method}})

Error above aside, I feel like I am missing something fundamental and that I took a wrong direction. Even if this worked, it feels wrong to introduce a new attribute with a nested structure to simply keep a method which avoids calls to this method as a class method with self in the front...

Maybe someone can suggest a better direction how to create a preferably serializable class which imitates a dictionary and that can also get parameters dynamically? An explanation of the error that I get would be very useful too, but I think this is not the biggest problem I am facing here. Thank you for any help in advance.

With a bit of tinkering, all these errors can be resolved. But I am confused as to why do this in the first place? Why not override the constructor for `Foo` so that it accepts argument `func` through which you can then save the `method_to_pass` as an attribute and call them from inside `special_method` by doing `self.func()`? By the way, you need to remove `abc.ABC` as the base class, it's reference is not serializable by either pickle or dill. — Charchit Agarwal, Jan 25 '23 at 08:19
Thank you for the comment, @CharchitAgarwal. I also feel that this is more complicated than it needs to be. The reason why I did this originally was to pass the customized type that the factory returns to `multiprocessing.manager.register` which expects a callable to create objects of the needed type: https://docs.python.org/3.6/library/multiprocessing.html#managers#register It could be that I can rework the factory and pass it here. — J.K., Jan 25 '23 at 09:00
A class is also a callable and will work with the `register` method. To me it seems like you could simply store the dynamic function in an attribute when you create an instance of the class through the manager and then call all your special functions. Something like `manager.register("Foo", Foo); foo = manager.Foo(*args, *kwargs, method_to_pass); foo.special_function(*args, *kwargs)`. Would this not be reasonable? — Charchit Agarwal, Jan 25 '23 at 09:10
@CharchitAgarwal, I made it work as you suggested! Thank you. Do you mind if I post what I ended up doing as an answer? — J.K., Jan 25 '23 at 16:39
I encourage you to :) To future readers who for some reason do need to serialize enclosed classes, check this [answer](https://stackoverflow.com/a/11526524/16310741) — Charchit Agarwal, Jan 25 '23 at 23:38

score 1 · Accepted Answer · answered Jan 26 '23 at 18:13

With a lot of help and encouragement from @CharchitAgarwal, I have figured out two solutions to the problem.

If you are looking for the solution to the error in my last attempt, then Pickle a dynamically parameterized sub-class can be consulted. Tried it, all worked with minimal alterations, but the final result was harder to follow and not satisfying. Due to ease of copy/pasting and the bitter taste afterwards, I will not share the solution here.
Better answer to the question is to stop for a moment and to reflect if you are on the right path. Had this doubt when posting the question and, fortunately, I am not committed to the structure above. I think it is better to use composition instead of inheritance. Code is shorter and much more simple. If necessary, one can bind the special parameters with functools.partial and objects are still going to be serializable in Python 3.6 that was used here.

import pickle


class Foo(dict):

    def __init__(self, *args, special_method=None, bar=None, **kwargs):
        super().__init__(*args, **kwargs)
        self.foo = 'foo'
        self._special_method = special_method
        self.bar = bar

    def special_method(self, arg1, arg2):
        print(self.foo, ' ', self.bar, ' ', self._special_method(arg1, arg2))


def method_to_pass(a1, a2):
    return a1 + a2


if __name__ == '__main__':
    foo = Foo({'foo': 'bar'}, special_method=method_to_pass, bar='bar')
    print(foo)
    foo.special_method(4, 2)
    pickle.dumps(foo)

Dynamically creating serializable classes in Python

1 Answers1