4

I am trying to use the multiprocessing package to call a function (let's call it myfunc) in parallel, specifically using pool.map i.e. pool.map(myfunc, myarglist). When I simply loop over myarglist without using multiprocessing there are no errors, which should be the case because all operations in myfunc are called within a try block. However, when I call the function using pool.map the script invariably stops running, i.e. it stop printing a "myfunc done!" statement within my function and the processes stop using the CPUs, but it never returns resultlist. I am running python 2.7 from the terminal in ubuntu 12.04. What could cause this to occur and how should I fix/troubleshoot the problem?

cpu_count = int(multiprocessing.cpu_count())
pool = Pool(processes = cpu_count)
resultlist = pool.map(myfunc, myarglist)
pool.close()

Update One issue when using multiprocessing can be the size of the object, if you think that may be a problem see this answer. As the answer notes "If this [solution] doesn't work, maybe the stuff you're returning from your functions is not pickleable, and therefore unable to make it through the Queues properly." Multiprocessing passes objects between processes by pickling them. It turns out that one or two of my objects had soup from BeautifulSoup that would not pickle.

Community
  • 1
  • 1
Michael
  • 13,244
  • 23
  • 67
  • 115
  • 3
    Impossible to know what's causing this without some information on the body of `myfunc`. – g.d.d.c Jul 02 '14 at 17:28
  • Can you tell if the function has run at all? – mgilson Jul 02 '14 at 17:29
  • Yes, I know the the function runs because it prints out "myfunc done!" over and over. I tried removing the return statement from `myfunc` and the pool.map() call completes. So it seems to be having troubles at the stage where pool.map combines the results. In looking at the system monitor though nothing is happening in any of the python processes at this point. And it is taking longer than it would to simply run on a single core. – Michael Jul 02 '14 at 17:38
  • 1
    removing the return makes the problem go away - is myarglist large or are you returning a particularly large object or one that has references to other objects? You may want to try map_async and use the callback to track how much gets done before all goes wrong. – tdelaney Jul 02 '14 at 19:11
  • So I have isolated an object that will not return when myfunc runs in parallel, but has no problem when myfunc is called directly. The object in question is about five times larger than the typical object in the list, but is otherwise normal. When I use map_async the function executes fine, but `result.get()` hangs. Interestingly(?), if and only if I call `myfunc` on that object not in parallel right before I call myfunc on the object using `map_async` I get the error message "Segmentation fault (core dumped)" before I even call `result.get()`. – Michael Jul 02 '14 at 20:15
  • myfunc is a lot of code and calls several other functions. I am not going to post it here, but everything is done within a `try` statement and function itself does not seem to be the problem, but returning the function's result in parallel is. – Michael Jul 02 '14 at 20:17
  • @Michael Can you say more about what the large object actually looks like? I'm assuming it's a custom class of some kind? What's in it? What makes it 5x larger than the other objects you're working with? Based on your comment about the segfault, I'm wondering if your child process is segfaulting at some point while executing `myfunc` or trying to return from `myfunc`. If that is indeed happening, `map` would hang forever. – dano Jul 03 '14 at 00:32
  • @Michael Also, is the object you're talking about the object being returned from `myfunc`, or the object being passed to `myfunc`? – dano Jul 03 '14 at 00:37

1 Answers1

6

Check whether all the processes are started or not.This will help you to debug it.Also add Pool.join() at the end of your code.

This is a sample code

def start_process():
    print 'Starting', multiprocessing.current_process().name

if __name__ == '__main__':

    pool_size =2
    pool = multiprocessing.Pool(processes=pool_size,
                                initializer=start_process,
                                )

    pool_outputs = pool.map(function_name,argument_list)
    pool.close() # no more tasks
    pool.join()  # wrap up current tasks
vicknick
  • 110
  • 7