Can't access attribute from class on multiprocessing proxy

Question

I have a custom class in which I need to access a protected attribute on my object of this class, the processing takes too long, so I want to parallelize it, and the data is too large, so I have to do it in a way that the processes share the object instead of copying it around.

I found out about managers on multiprocessing, and I started writing one for my class which led me to an error similar to the one below:

AttributeError: 'AutoProxy[Test]' object has no attribute '_foo'

So I did some research and tried following the code in here which led me to a code similar to the one below:

from multiprocessing.managers import BaseManager, NamespaceProxy
from random import random, randint
import multiprocessing as mp



class TestClass:
    def __init__(self):
        self._foo = {key: random() for key in range(100)}


class TestManager(BaseManager):
    pass


class TestProxy(NamespaceProxy):
    _exposed_ = ('__getattribute__', '__setattr__', '__delattr__')


TestManager.register('Test', TestManager, TestProxy)


def print_random(bar):
    print(bar._foo[randint(0, 99)])


if __name__ == '__main__':
    manager = TestManager()
    manager.start()
    test = manager.Test()

    pool = mp.Pool(mp.cpu_count())
    for _ in range(100):
        pool.apply(print_random, args=(test,))

    pool.close()
    pool.join()

But I'm still the following error, and after searching a lot I could not find a way of getting around it:

AttributeError: 'TestProxy' object has no attribute '_foo'

Any idea of what could be happening and how I should fix it?

Answer 1

I spot two mistakes. Firstly, you need to register TestClass instead of TestManager when you are registering a proxy. Therefore the line:

TestManager.register('Test', TestManager, TestProxy) should become:

TestManager.register('Test', TestClass, TestProxy)

Next, you are trying to access a protected variable from outside the class. Even though Python allows you to do that, it doesn't make sense for it to be protected then. But more importantly, the proxy object created by NamespaceProxy class doesn't share protected attributes and hence it is not accessible through the instance of TestClass.

Hence, you should either make print_random() a member function of TestClass, or make the attribute foo public. Doing the latter like below works without issues:

from multiprocessing.managers import BaseManager, NamespaceProxy
from random import random, randint
import multiprocessing as mp



class TestClass:
    def __init__(self):
        self.foo = {key: random() for key in range(100)}


class TestManager(BaseManager):
    pass


class TestProxy(NamespaceProxy):
    _exposed_ = ('__getattribute__', '__setattr__', '__delattr__')


TestManager.register('Test', TestClass, TestProxy)


def print_random(bar):
    print(bar.foo[randint(0, 99)])


if __name__ == '__main__':
    manager = TestManager()
    manager.start()
    test = manager.Test()

    pool = mp.Pool(mp.cpu_count())
    for _ in range(100):
        pool.apply(print_random, args=(test,))

    pool.close()
    pool.join()

Other than that, if you don't want to make the attribute public and can't access it from within the class (or subclass), then you can provide getter and setter methods, expose them in the proxy, and get the value of the protected attribute from there like below:

from multiprocessing.managers import BaseManager, NamespaceProxy
from random import random, randint
import multiprocessing as mp



class TestClass:
    def __init__(self):
        self._foo = {key: random() for key in range(100)}

    def get_foo(self):
        return self._foo


class TestManager(BaseManager):
    pass


class TestProxy(NamespaceProxy):
    _exposed_ = ('__getattribute__', '__setattr__', '__delattr__', 'get_foo')

    def get_foo(self):
        callmethod = object.__getattribute__(self, '_callmethod')
        return callmethod('get_foo', ())


TestManager.register('Test', TestClass, TestProxy)


def print_random(bar):
    print(bar.get_foo()[randint(0, 99)])


if __name__ == '__main__':
    manager = TestManager()
    manager.start()
    test = manager.Test()

    pool = mp.Pool(mp.cpu_count())
    for _ in range(100):
        pool.apply(print_random, args=(test,))

    pool.close()
    pool.join()