Using sets with the multiprocessing module

Question

I can't seem to share a set across processes using a Manager instance. A condensed version of my code:

from multiprocessing.managers import SyncManager
manager = SyncManager()
manager.start()
manager.register(Set)

I've also tried register(type(Set)) and register(Set()), but I'm not overly surprised that neither of them worked (the first should evaluate to Class, I think).

The exception I get in all cases is TypeError: __name__ must be set to a string object in line 675 of managers.py.

Is there a way of doing this, or do I need to investigate alternatives?

Why use `Set` (the module) when you have `set` (the built-in type)? — Martijn Pieters, May 07 '13 at 09:04
And the `.register()` method needs more arguments, the first argument being the *name* of the type. — Martijn Pieters, May 07 '13 at 09:05

Martijn Pieters · Accepted Answer · 2013-05-07T09:34:23.177

The first argument to the SyncManager.register() class method must be a string, not a cass:

SyncManager.register('set', set)

but you'll need to register a proxy for sets as well:

from multiprocessing.managers import MakeProxyType

BaseSetProxy = MakeProxyType('BaseSetProxy', (
    '__and__', '__contains__', '__iand__', '__ior__', 
    '__isub__', '__ixor__', '__len__', '__or__', '__rand__', '__ror__', '__rsub__',
    '__rxor__', '__sub__', '__xor__', 'add', 'clear', 'copy', 'difference',
    'difference_update', 'discard', 'intersection', 'intersection_update', 'isdisjoint',
    'issubset', 'issuperset', 'pop', 'remove', 'symmetric_difference', 
    'symmetric_difference_update', 'union', 'update']
    ))
class SetProxy(BaseSetProxy):
    # in-place hooks need to return `self`, specify these manually
    def __iand__(self, value):
        self._callmethod('__iand__', (value,))
        return self
    def __ior__(self, value):
        self._callmethod('__ior__', (value,))
        return self
    def __isub__(self, value):
        self._callmethod('__isub__', (value,))
        return self
    def __ixor__(self, value):
        self._callmethod('__ixor__', (value,))
        return self

SyncManager.register('set', set, SetProxy)

Of course! Thanks for pointing out the stuff I really should have noticed when I RTFM. — Hotchips, May 07 '13 at 09:33
@Hotchips: I am re-introducing the `__i*__` names to the `exposed` list for the proxy; reading the `ListProxy` code I think I may have discovered a bug in that it doesn't expose the `list.__iadd__` method properly. Not 100%, but listing them in the `BaseSetProxy` list does no harm, and omitting them could be a bug. — Martijn Pieters, May 07 '13 at 09:35

Using sets with the multiprocessing module

1 Answers1

Linked