Python Objects in Multiprocessing

Question

I was testing python multiprocessing with a simple program:

 class MyTest:
    def __init__(self):
        self.myattr = 1

    def myfunc(self):
        print(self)
        self.myattr=2
        print(self)
        print("in proc"+str(self.myattr))



def main():

    test = MyTest()
    print(test)
    myProc = Process(target = test.myfunc,args=())

    myProc.start()
    myProc.join()
    print(test.myattr)

main()

The output of self in the method that the process is running has the same address as that of the object in main, which I found quite strange since they are separate processes. However, since they are at the same address, they are in fact shared across processes and updating the attribute myattr in the myfunc should change the attr of test in main. However print(test.myattr) still prints 1 even after the process has finished updating test's attribute.

My first question is why do these two objects in separate processes share the same address, and my second is if they do share the same address why aren't changes in one process recognized by changes in another?

P.S. does my first question have something to do with the fact that I am passing the process a object method to run as its target? i.e. a object function call such as test.myfunc = MyTest.myfunc(test).

The output:

<__main__.MyTest object at 0x7f007bdf76d8>
<__main__.MyTest object at 0x7f007bdf76d8>
<__main__.MyTest object at 0x7f007bdf76d8>
in proc2
1

I think the answer might be here: http://stackoverflow.com/questions/20955683/python-multiprocessing-sharing-a-complex-object and here: http://stackoverflow.com/a/33701697/289011 (particularly, the part that says _address pointers are virtual, therefore they represent an offset within the process address space itself_ ) — Savir, Oct 25 '16 at 14:33
So it must be that coincidentally, the main process and the child process have the original object (main process) and the copied object (child process) at the same offset in their coresponding Virtual address spaces? — dylan7, Oct 25 '16 at 20:16
That must be it... I've gotta say, this puzzled me too **:-S** It's a nice question (at least it helped me learn something that I thought it'd work in a different way) — Savir, Oct 25 '16 at 20:25

score 0 · Answer 1 · edited May 23 '17 at 12:09

I think you should use pathos, it simplifies multiprocessing with classes and works like a breeze.

Step 1:

pip install pathos

Step 2:

Refactor your code to be able to run without an __init__(), then have a class method that runs what you want to run, the instance variables should be in the class method that you want run.

Step 3:

Then run it however you want.

Example:

from pathos.multiprocessing import ProcessingPool as Pool 
p = Pool(4)
class MyClass:

  def add(self, x, y):
      return x+y

mc = MyClass()
x = [1,2,3,4,5]
y = [0,2,4,6,8]
result = p.amap(mc.add, x, y)
print result.get()

Source: https://stackoverflow.com/a/21345308/7372029

https://github.com/uqfoundation/pathos/tree/master/examples

Python Objects in Multiprocessing

1 Answers1