Accessing a subclass method of a multiprocessing Proxy of a class

Question

I have a class being managed by a multiprocessing manager. I am able to expose it's methods and have seen posts on how to expose it's attributes

Accessing an attribute of a multiprocessing Proxy of a class

I am wondering how to call methods from a subclass attribute of a managed class like seen bellow

from multiprocessing.managers import BaseManager

class TestSubClass:
    def __init__(self):
        self.a = 1
    def set_a(self, a):
        self.a = a

class TestClass:
    def __init__(self):
        self.subclass = TestSubClass()
    def add_to_a(self, b):
        return b + self.subclass.a

class MyManager(BaseManager): pass

MyManager.register('TestClass', TestClass)

if __name__ == '__main__':

    with MyManager() as manager:

        # Creates classes
        t = TestClass()
        mt : TestClass = manager.TestClass()

        # Runs the Exposed add_to_a method
        print (t.add_to_a(2))
        print (mt.add_to_a(2))

        # Runs the subclass method for the regular class
        t.subclass.set_a(2)
        print (t.add_to_a(2))

        # Runs the subclass method for the proxy class (Fails)
        mt.subclass.set_a(2)
        print (mt.add_to_a(2))

The key difference from the above linked question is that I'm not trying to return the subclass to the main process but rather simply call one of it's methods in the manager's process

You may want to be careful with your terminology here. You're using "subclass" to mean something very different than what that term normally means. It's not related to inheritance here, as far as I can see. — Blckknght, Apr 08 '21 at 23:30
Good point, I agree subclass may not be best. Do you have a term to recommend? I'll make an edit — Andrew, Apr 08 '21 at 23:34

Booboo · Accepted Answer · 2021-04-18T15:10:25.160

You need to define a proxy class for your TestClass that supports attributes. This should inherit from NamespaceProxy to support attributes and you will have to explicitly expose the methods as follows:

class TestClassProxy(NamespaceProxy):
    _exposed_ = ('__getattribute__', '__setattr__', '__delattr__', 'add_to_a')

    def add_to_a(self, b):
        callmethod = object.__getattribute__(self, '_callmethod')
        return callmethod('add_to_a', args=(b,))

I would also suggest renaming TestSubClass to TestContainedClass. And although it doesn't matter in your code as it currently stands, I would move the register statement to be within the if __name__ == '__main__': block so that if this code is ever modified to create processes, the register statement is not needlessly executed by the newly created processes if you happen to be on a platform that uses spawn to create these processes (e.g. Windows). It's not a big deal one way or another.

Putting this all together:

from multiprocessing.managers import BaseManager, NamespaceProxy

class TestContainedClass:
    def __init__(self):
        self.a = 1
    def set_a(self, a):
        self.a = a

class TestClass:
    def __init__(self):
        self.subclass = TestContainedClass()
    def add_to_a(self, b):
        return b + self.subclass.a

class MyManager(BaseManager): pass

class TestClassProxy(NamespaceProxy):
    _exposed_ = ('__getattribute__', '__setattr__', '__delattr__', 'add_to_a')

    def add_to_a(self, b):
        callmethod = object.__getattribute__(self, '_callmethod')
        return callmethod('add_to_a', args=(b,))



if __name__ == '__main__':

    MyManager.register('TestClass', TestClass, TestClassProxy)

    with MyManager() as manager:

        # Creates classes
        t = TestClass()
        mt : TestClass = manager.TestClass()

        # Runs the Exposed add_to_a method
        print (t.add_to_a(2))
        print (mt.add_to_a(2))

        # Runs the subclass method for the regular class
        t.subclass.set_a(2)
        print (t.add_to_a(2))

        # Runs the subclass method for the proxy class (Fails)
        mt.subclass.set_a(2)
        print (mt.add_to_a(2))

Prints:

But notice the last result is 3 and not the expected 4. Nested objects does not seem to work well with the Manager. See Sharing a complex object between processes?.

To get this to work, both classes must be represented by proxies:

from multiprocessing.managers import BaseManager, NamespaceProxy

class MyManager(BaseManager): pass

class TestContainedClass:
    def __init__(self):
        self.a = 1
    def set_a(self, a):
        self.a = a

class TestContainedClassProxy(NamespaceProxy):
    _exposed_ = ('__getattribute__', '__setattr__', '__delattr__', 'set_a')

    def set_a(self, a):
        callmethod = object.__getattribute__(self, '_callmethod')
        return callmethod('set_a', args=(a,))

class TestClass:
    def __init__(self, subclass):
        self.subclass = subclass
    def add_to_a(self, b):
        return b + self.subclass.a


class TestClassProxy(NamespaceProxy):
    _exposed_ = ('__getattribute__', '__setattr__', '__delattr__', 'add_to_a')

    def add_to_a(self, b):
        callmethod = object.__getattribute__(self, '_callmethod')
        return callmethod('add_to_a', args=(b,))


if __name__ == '__main__':

    MyManager.register('TestContainedClass', TestContainedClass, TestContainedClassProxy)
    MyManager.register('TestClass', TestClass, TestClassProxy)

    with MyManager() as manager:

        # Creates classes
        testContainedClass = manager.TestContainedClass()
        mt : TestClass = manager.TestClass(testContainedClass)

        # Runs the Exposed add_to_a method
        print (mt.add_to_a(2))

        # Runs the subclass method for the proxy class
        mt.subclass.set_a(2)
        print (mt.add_to_a(2))

Prints:

3
4

It would, of course, be simpler, just to refactor your original code:

from multiprocessing.managers import BaseManager, NamespaceProxy


class TestClass:
    def __init__(self, a):
        self.a = a
    def add_to_a(self, b):
        return b + self.a

class MyManager(BaseManager): pass

class TestClassProxy(NamespaceProxy):
    _exposed_ = ('__getattribute__', '__setattr__', '__delattr__', 'add_to_a')

    def add_to_a(self, b):
        callmethod = object.__getattribute__(self, '_callmethod')
        return callmethod('add_to_a', args=(b,))


if __name__ == '__main__':

    MyManager.register('TestClass', TestClass, TestClassProxy)

    with MyManager() as manager:

        # Creates classes
        t = TestClass(1)
        mt : TestClass = manager.TestClass(1)

        # Runs the Exposed add_to_a method
        print (t.add_to_a(2))
        print (mt.add_to_a(2))

        # Runs the subclass method for the regular class
        t.a = 2
        print (t.add_to_a(2))

        # Runs the subclass method for the proxy class
        mt.a = 2
        print (mt.add_to_a(2))

Prints:

Or, if you are going to access attribute a using method set_a, then you can use the default proxy:

from multiprocessing.managers import BaseManager, NamespaceProxy


class TestClass:
    def __init__(self, a):
        self.set_a(a)
    def set_a(self, a):
        self.a = a
    def add_to_a(self, b):
        return b + self.a

class MyManager(BaseManager): pass



if __name__ == '__main__':

    MyManager.register('TestClass', TestClass)

    with MyManager() as manager:

        # Creates classes
        t = TestClass(1)
        mt : TestClass = manager.TestClass(1)

        # Runs the Exposed add_to_a method
        print (t.add_to_a(2))
        print (mt.add_to_a(2))

        # Runs the subclass method for the regular class
        t.set_a(2)
        print (t.add_to_a(2))

        # Runs the subclass method for the proxy class
        mt.set_a(2)
        print (mt.add_to_a(2))

Accessing a subclass method of a multiprocessing Proxy of a class

1 Answers1