Factory method pattern conflicts with use of multiprocessing queue

Question

I have implemented the factory method pattern to parametrize the base class of the product class:

def factory(ParentClass):
    class Wrapper(ParentClass):
        _attr = "foo"

        def wrapped_method():
            "Do things to be done in `ParentClass`."""
            return _attr

    return Wrapper

I need to share Wrapper objects with a process spawned using the multiprocessing module by means of a multiprocessing.Queue.

Since multiprocessing.Queue uses Pickle to store the objects (see note at Pickle documentation), and Wrapper is not defined at the top level, I get the following error:

PicklingError: Can't pickle <class 'Wrapper'>: attribute lookup Wrapper failed

I used the workaround in this answer and I get another error:

AttributeError: ("type object 'ParentClass' has no attribute 'Wrapper'", <main._NestedClassGetter object at 0x8c7fe4c>, (<class 'ParentClass'>, 'Wrapper'))

Is there a solution to share these sort of objects among processes?

The workaround isn't working because `Wrapper` isn't nested inside `ParentClass`, its nested inside the `factory` function. — dano, Jul 09 '14 at 15:34

score 1 · Accepted Answer · answered Jul 09 '14 at 16:06

According to the Pickle documentation, the workaround linked in the question could be modified to:

class _NestedClassGetter(object):
    """
    From: http://stackoverflow.com/a/11493777/741316
    When called with the containing class as the first argument,
    and the name of the nested class as the second argument,
    returns an instance of the nested class.
    """
    def __call__(self, factory_method, base):
        nested_class = factory_method(base)

        # make an instance of a simple object (this one will do), for which we
        # can change the __class__ later on.
        nested_instance = _NestedClassGetter()

        # set the class of the instance, the __init__ will never be called on
        # the class but the original state will be set later on by pickle.
        nested_instance.__class__ = nested_class
        return nested_instance

and the __reduce__ method to:

    def __reduce__(self):
        state = self.__dict__.copy()
        return (_NestedClassGetter(),
                (factory, ParentClass), state,)

Thanks to @dano for his comment.

Jamie Cockburn · Answer 2 · 2014-07-10T12:14:24.177

1

The best solution is to restructure your code to not have dynamically declared classes, but assuming that isn't the case, you can do a little more work to pickle them.

And this method to your Wrapper class:

def __reduce__(self):
    r = super(Wrapper, self).__reduce__()
    return (wrapper_unpickler, 
            ((factory, ParentClass, r[0]) + r[1][1:])) + r[2:]

Add this function to your module:

def wrapper_unpickler(factory, cls, reconstructor, *args):
    return reconstructor(*((factory(cls),) + args))

Essentially, you are swapping the dynamically generated Wrapper class for the factory funciton + wrapped class when pickling, and then when unpickling, dynamically generating the Wrapper class again (passing the wrapped type to the factory) and swapping the wrapped class for the Wrapper.

edited Jul 10 '14 at 12:14

answered Jul 09 '14 at 16:26

Jamie Cockburn

7,379
1
24
37

But as I said in the question: my goal *is* to use dynamically declared classes, as I want to use the factory method pattern. In which way do you propose to structure the code in order to have the same functionality provided by this pattern? – synack Jul 10 '14 at 09:23
@Kits89 That's why I said "but assuming that isn't the case", meaning "assuming that's not possible". The rest of the answer then gives you a working solution to your problem! – Jamie Cockburn Jul 10 '14 at 09:28
I was just curious about whether there was an equivalent pattern. I'd rather not to use dynamically generated classes. Yes, thanks for your answer, it's pretty much the same thing I came up with in the end (see the other answer). Thanks for your help :) – synack Jul 10 '14 at 09:46
@Kits89 It occured to me I could do it without the extra import. (See my edit.) Without knowing *why* you're creating the classes dynamically like you are, it's hard to suggest an alternative, but for the example you've given your `factory` method could just contain `setattr(ParentClass, "wrapped_method", wrapped_method)` to directly add it to the class. – Jamie Cockburn Jul 10 '14 at 12:11
@Kits89 is there any reason you can't just make Wrapper a base of your statically defined classes? – Jamie Cockburn Jul 10 '14 at 12:13
Well, in fact the statically defined classes are already children of a common parent class. I could just add the wrapper methods to the already existing base class. The thing is that these methods are just intended for testing purposes and I think is kind of ugly to add them in the base class. That's why I was thinking in a wrapper. Nice edit btw! – synack Jul 10 '14 at 12:20
1

@Kits89 Well then you can just monkey-patch them onto that base class: `setattr(Base, "method_name", method)` in your test code. – Jamie Cockburn Jul 10 '14 at 12:22
@Kits89 of course, since nothing in python is "private", why can't you just have a regular function in your test code? Is there a real need to be able to call that function as a method? – Jamie Cockburn Jul 10 '14 at 12:27
I have no access to the class in runtime. The class is part of a `twisted` based module that runs in a separate process. The idea is to provide methods to the class that dump its state to disk and then load them in my test methods. – synack Jul 10 '14 at 12:32
Also, some of the methods of the wrapper need to change the behaviour of the base class... :S – synack Jul 10 '14 at 12:45
As in they are overrides? Then monkey-patching is the way to go! – Jamie Cockburn Jul 10 '14 at 12:46
hmm, sorry, so how do I override a method by monkey-patching? – synack Jul 10 '14 at 12:50

Factory method pattern conflicts with use of multiprocessing queue

2 Answers2

Linked