3

I have a class that contains a large member variable. I want some computation to be applied to this variable concurrently.

class A(object):
    def __init__(self):
      self.x = numpy.random.random((1e6,)) # some really large array
    def func(self, t): # a function that is expected to run concurrently
      # some computation that uses self.x
      return 0 # here I just return 0 for simplicity

I call the class this way (using pathos's multiprocess):

from multiprocess import Pool
a = A()
pool = Pool(16)
results = pool.map(a.func, range(16))

It is very slow. I suspect it is because the large array is copied to every process, instead of being shared. How can I share this large class member variable between the processes?

P.S. I used pathos's multiprocess instead of Python's multiprocessing because the latter cannot pickle instance method.

mistycheney
  • 33
  • 4
  • Yes, the class is copied to each process, so the array is copied. You may be able to use a shared memory array. See: http://stackoverflow.com/q/1675766/2379433 and http://stackoverflow.com/q/7894791/2379433 – Mike McKerns Jun 10 '16 at 11:36

0 Answers0