When does multiprocessing.BaseManager use RemoteError?

Question

I have a C extension module for CPython. I need multiple uWSGI workers to share a single instance of an object in this module. I am using a custom subclass of multiprocessing.BaseManager to accomplish this, based on this answer that describes a very similar solution.

The first script below is wifi.manager (wifi.controller.IFace is the object to be shared). I run this with python3 wifi/manager.py and then start the web server, which runs the code in the second snippet to get the shared object instance.

wifi/manager.py:

#!/usr/bin/env python3

from multiprocessing.managers import BaseManager

# .register() changes the class itself. We don't want to do that to BaseManager.
class WifiManager(BaseManager):
    pass

if __name__ == '__main__':
    # If we are executed as a script (python3 manager.py), start the server
    import atexit
    from multiprocessing import Lock
    import wifi.controller

    ifaces_lock = Lock()
    ifaces = dict()

    def get_iface(iface_path):
        with ifaces_lock:
            if iface_path not in ifaces:
                # Control interface isn't open. Open it.
                iface = wifi.controller.IFace(iface_path)
                ifaces[iface_path] = iface
        return ifaces[iface_path]

    def close_ifaces():
        for iface in ifaces.values():
            iface.close()

    WifiManager.register('get_iface', get_iface)
    atexit.register(close_ifaces)

    manager = WifiManager(address=('127.0.0.1', 2437), authkey=b'wifimanager')
    server = manager.get_server()
    server.serve_forever()
else:
    # If we are imported, provide the WifiManager class ready for clients to use
    WifiManager.register('get_iface')

Snippet from web application:

from wifi.manager import WifiManager

...

wmanager = WifiManager(address=('127.0.0.1', 2437), authkey=b'wifimanager')
wmanager.connect()
iface = wmanager.get_iface(iface_path)

iface.scan() # And other code using the iface object

The wifi.controller.IFace object sometimes raises exceptions, either built-in ones (mainly OSError) or its own wifi.controller.WifiError exception. Sometimes, I want to be able to catch these in the web app to present meaningful error pages to the client. But, what I've noticed is sometimes, these exceptions get captured and the same exception (e.g. WifiError) is raised in the web app. Other times, the web app gets a multiprocessing.managers.RemoteError with the traceback from the manager stored as a string.

The question is, how do I know when it will raise the original exception and when it will raise a RemoteError, so I know which one to catch? All the Python docs say is this:

If an exception is raised by the call, then is re-raised by _callmethod(). If some other exception is raised in the manager’s process then this is converted into a RemoteError exception and is raised by _callmethod().

This isn't very clear to me and I can't figure out how it lines up with the behavior I've observed.

score 0 · Accepted Answer · answered May 03 '19 at 14:03

I believe I've figured it out. I'm still not 100% sure, but seeing as how this question isn't getting a lot of attention, I figured I'll go ahead and add an answer in case future readers stumble across this.

Exceptions that happen in the WifiManager during the original remote call get raised as RemoteError. In this case, that means exceptions on the remote end during this line in the web app:

iface = wmanager.get_iface(iface_path)

After that, the web app is no longer directly interacting with the WifiManager. It is only interacting with iface, a proxy object. If a proxy object (or more accurately, its referent) raises an exception on the remote side, the same exception gets raised in the web app, not a RemoteError. So, that means if this line were to cause e.g. a WifiError or OSError, those are the exceptions you would catch:

iface.scan() # And other code using the iface object

So, to summarize, calls on BaseManager (or subclasses) raise RemoteError when there is an exception on the remote end. Calls on proxy objects raise a copy of the original exception.

Bear in mind, this answer is based on my observations, not my complete and total understanding of the situation, so it's possible there are some gotchas here I'm unaware of. If somebody knows better, please correct me. But this describes the behavior I've observed and seems consistent with the docs quoted in the question.

When does multiprocessing.BaseManager use RemoteError?

1 Answers1