4

i'm trying to use multiprocessing.Pool to parallelize an operation on a set of class objects.

obj= []
for i in range(20):
    obj.append(myClass(i))
pool= multiprocessing.Pool(processes=4)
pool.map(do_something, obj)
pool.map(do_something_else, obj)

and myClass and its function do_something and do_something_else are like this..

class myClass:
    def __init__(self,i):
        self.obj_id= i
        self.value= 0

    def do_something():
        self.value = self.value + 1   #some operations

    def do_something_else():
        self.value = self.value * 99   #some operations

How can i do this parallelization? and will the 'value' of the original objects in obj get updated? Can anyone help me with a solution? Thanks in advance.

Michael Foukarakis
  • 39,737
  • 6
  • 87
  • 123
sumanth
  • 43
  • 4
  • possible duplicate of [Can't pickle when using python's multiprocessing Pool.map()](http://stackoverflow.com/questions/1816958/cant-pickle-type-instancemethod-when-using-pythons-multiprocessing-pool-ma) – Michael Foukarakis Mar 26 '14 at 06:50

1 Answers1

0

The map function does not call methods on a list of objects, but a function with each object as first argument.

Therefore you need a helper function:

def _call_do_something(obj):
    obj.do_something ()
    return obj

# ...
obj_list = pool.map(_call_do_something, obj_list)

Since this will spawn a new process that needs to find _call_do_something via import (at least on Windows), make it a module-level function.

Ferdinand Beyer
  • 64,979
  • 15
  • 154
  • 145
  • 1
    This won't work, as the object state won't be transferred back to the main process. At the main process the value of `obj.value` will still be 0 – justhalf Mar 26 '14 at 06:20
  • i tried the same thing with a helper function. But i got an error saying "TypeError: object of type 'int' has no len() ". may be its a problem with the iterable. and as @justhalf mentioned, obj.value won't be updated. – sumanth Mar 26 '14 at 06:28
  • 1
    No, you'll get a new object with an updated value. Your error is caused because somewhere you are passing an integer where a sequence is expected, like `len(42)` or `map(func, 23)`. The traceback tells you where. – Ferdinand Beyer Mar 26 '14 at 06:54
  • 1
    ok, i rectified my error. But, still the obj.value won't be updated. Oh, i didn't see that reassignment to obj_list. Cool. Using that return and reassignment, one can get the updated obj. Worked for me..Thank you @ferdinand – sumanth Mar 26 '14 at 07:08