multiprocessing.Manager nested shared objects doesn't work with Queue

Question

Python docs of the multiprocessing module state:

Changed in version 3.6: Shared objects are capable of being nested. For example, a shared container object such as a shared list can contain other shared objects which will all be managed and synchronized by the SyncManager.

This does work with list and dict. However, if I try to create a shared Queue inside a shared dict, I get an error:

>>> from multiprocessing import Manager
>>> m = Manager()
>>> d = m.dict()
>>> d['a'] = m.list()
>>> d['b'] = m.dict()
>>> d['c'] = m.Queue()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 2, in __setitem__
  File "/usr/lib/python3.6/multiprocessing/managers.py", line 772, in _callmethod
    raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError: 
---------------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/managers.py", line 228, in serve_client
    request = recv()
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 251, in recv
    return _ForkingPickler.loads(buf.getbuffer())
  File "/usr/lib/python3.6/multiprocessing/managers.py", line 881, in RebuildProxy
    return func(token, serializer, incref=incref, **kwds)
TypeError: AutoProxy() got an unexpected keyword argument 'manager_owned'
---------------------------------------------------------------------------

Seems like https://hg.python.org/cpython/rev/39e7307f9aee is the changeset which introduced nested shared objects.

score 5 · Accepted Answer · answered Jun 23 '19 at 02:25

5

The error is caused by AutoProxy currently not handling all BaseProxy arguments. There's a pull request which has not been merged yet. You either need to monkey-patch AutoProxy, or you look into multiprocessing.managers.py and apply the changes in the patch here directly to your source code.

It's really important to fix both lines in the patch to prevent a memory leak in the server process. The manager_owned-flag is used to let the BaseProxy code know, when to skip a reference increment for a proxy the manager owns himself (through nesting).

answered Jun 23 '19 at 02:25

Darkonaut

20,186
7
54
65

Is this solution sufficient, or does it risk a memory leak? https://stackoverflow.com/questions/46779860/multiprocessing-managers-and-custom-classes – David Parks Oct 31 '19 at 02:45
1

@DavidParks I had a play with the source code before I wrote this answer and the answer (both) you linked is not sufficient as it currently is. Created objects will always have a reference in the manager-process and hence never be destroyed. – Darkonaut Oct 31 '19 at 10:31
Thanks, when you say created objects, you mean objects created by the manager? If so, then if we create a fixed number of objects (say a few manager.Queue objects) then it sounds like it's not an issue, right? Or do you mean every object that is added to a manager.Queue, or other manager data structures, will leave a residue? – David Parks Oct 31 '19 at 17:08
1

@DavidParks Sorry, with "objects" I meant only the nested queues itself, other objects you put inside it should not be affected. So unless you create masses of nested queues, it's not going to be a big issue. But the additional change to make it a clean solution is small enough, so I would just implement the patch as a whole and tick it off forever. – Darkonaut Oct 31 '19 at 18:05
1

For what it’s worth, I created a monkey patch that handles this all properly (handling the `manager_owned` flag properly and re-registering the existing SyncManager registrations that use AutoProxy). See [my answer on the duplicate post](https://stackoverflow.com/a/63797696/100297). – Martijn Pieters Sep 08 '20 at 22:48

David Parks · Answer 2 · 2019-11-02T15:40:25.977

If you don't want to patch the underlying python library, you can apply this patch in your own code using the following.

I've copied the change from the pull request reference by @Darkonaut and made package name modifications so it works outside the original package. This is placed at the module level of any module that uses multiprocessing.managers.

Note that the solution I referenced in comments in @Darkonaut's answer produced seg faults in my own testing, but this solution did not.

import multiprocessing.managers

def AutoProxy(token, serializer, manager=None, authkey=None,
              exposed=None, incref=True, manager_owned=False):
    '''
    Return an auto-proxy for `token`
    '''
    _Client = multiprocessing.managers.listener_client[serializer][1]

    if exposed is None:
        conn = _Client(token.address, authkey=authkey)
        try:
            exposed = dispatch(conn, None, 'get_methods', (token,))
        finally:
            conn.close()

    if authkey is None and manager is not None:
        authkey = manager._authkey
    if authkey is None:
        authkey = multiprocessing.process.current_process().authkey

    ProxyType = multiprocessing.managers.MakeProxyType('AutoProxy[%s]' % token.typeid, exposed)
    proxy = ProxyType(token, serializer, manager=manager, authkey=authkey,
                      incref=incref, manager_owned=manager_owned)
    proxy._isauto = True
    return proxy

multiprocessing.managers.AutoProxy = AutoProxy

In W10_64, with Py3.7.7 adding this in the code raise an error when you try to run a `SyncManager`: `_pickle.PicklingError: Can't pickle : it's not the same object as multiprocessing.managers.AutoProxy`. It seems to work in WSL. — Matteo Ragni, Jul 02 '20 at 10:41
Patching the function is not enough. [My answer](https://stackoverflow.com/a/63797696/100297) to the older post this one is a duplicate of also re-registers all SyncManager types that used the unpatched AutoProxy, because otherwise you get pickling errors. — Martijn Pieters, Sep 08 '20 at 22:42

multiprocessing.Manager nested shared objects doesn't work with Queue

2 Answers2

Linked