Python multiprocessing with pathos

Question

I am trying to use Python's pathos to designate computations into separate processes in order to accelerate it with multicore processor. My code is organized like:

class:
   def foo(self,name):
    ...
    setattr(self,name,something)
    ...
   def boo(self):
      for name in list:
         self.foo(name)

As I had pickling problems with multiprocessing.Pool, I decided to try pathos. I tried, as suggested in previous topics:

import pathos.multiprocessing

but it resulted in error: No module multiprocessing - which I can't find in latest pathos version.

Then I tried modify boo method:

def boo(self):
 import pathos
 pathos.pp_map.pp_map(self.foo,list)

Now there is no error thrown, but foo does not work - instance of my class has no new attributes. Please help me, because I have no idea where to move next, after a day spent on that.

Answer 1

I'm the pathos author. I'm not sure what you want to do from your code above. However, I can maybe shed some light. Here's some similar code:

>>> from pathos.multiprocessing import ProcessingPool
>>> class Bar:
...   def foo(self, name):
...     return len(str(name))
...   def boo(self, things):
...     for thing in things:
...       self.sum += self.foo(thing)
...     return self.sum
...   sum = 0
... 
>>> b = Bar()
>>> results = ProcessingPool().map(b.boo, [[12,3,456],[8,9,10],['a','b','cde']])
>>> results
[6, 4, 5]
>>> b.sum
0

So what happens above, is that the boo method of the Bar instance b is called where b.boo is passed to a new python process, and then evaluated for each of the nested lists. You can see that the results are correct… len("12")+len("3")+len("456") is 6, and so on.

However, you can also see that when you look at b.sum, it's mysteriously still 0. Why is b.sum still zero? Well, what multiprocessing (and thus also pathos.multiprocessing) does, is make a COPY of whatever you pass through the map to the other python process… and then the copied instance is then called (in parallel) and return whatever results are called by the method invoked. Note you have to RETURN results, or print them, or log them, or send them to a file, or otherwise. They can't go back to the original instance as you might expect, because it's not the original instance that's sent over to the other processors. The copies of the instance are created, then disposed of -- each of them had their sum attribute increased, but the original `b.sum' is untouched.

There is however, plans within pathos to make something like the above work as you might expect -- where the original object IS updated, but it doesn't work like that yet.

EDIT: If you are installing with pip, note that the latest released version of pathos is several years old, and may not install correctly, or may not install all of the submodules. A new pathos release is pending, but until then, it's better to get the latest version of the code from github, and install from there. The trunk is for the most part stable under development. I think your issue may have been that not all packages were installed, due to a "new" pip -- "old" pathos incompatibility in the install. If pathos.multiprocessing is missing, this is the most likely culprit.

Get pathos from github here: https://github.com/uqfoundation/pathos

Answer 2

Here's how I go about this - I put the function to be run in parallel outside the class and pass the object as an arg while calling pool.map. Then, I return the object to be reassigned.

from pathos.multiprocessing import ProcessingPool


def boo(args):
    b, things = args
    for thing in things:
        b.sum += b.foo(thing)
    return [b, b.sum]

class Bar:
    def __init__(self):
       self.sum = 0
    def foo(self, name):
       return len(str(name))

pool = ProcessingPool(2)
b1 = Bar()
b2 = Bar()
print(b1, b2)

results = pool.map(boo, [[b1, [12,3,456]],[b2, ['a','b','cde']]])

b1, b1s = results[0]
b2, b2s = results[1]
print(b1,b1s,b1.sum)
print(b2, b2s, b2.sum)

Output:

(<__main__.Bar instance at 0x10b341518>, <__main__.Bar instance at 0x10b341560>)
(<__main__.Bar instance at 0x10b3504d0>, 6, 6)
(<__main__.Bar instance at 0x10b350560>, 5, 5)

Note that b1 and b2 are no longer the same as what they were before calling map because copies of them were made to be passed, as described by @Mike McKerns. However, the values of all their attributes are intact because they were passed, returned and reassigned.