Multiprocessing works with standalone method but not with class instance method

Question

I use multiprocessing lib to test python multi process, but I meet some problems. I have test code 1:

import multiprocessing

def test(name):
    print 'processing.....'
    tmp = 0
    for i in xrange(1000000000):
        tmp += i
    print 'process done'

if __name__ == '__main__':

    pools = multiprocessing.Pool()
    for i in xrange(2):
        pools.apply_async(test)
    pools.close()
    pools.join()

result is:

processing
processing
done
done

Code 2:

import multiprocessing

class Test:
    def test(name):
        print 'processing.....'
        tmp = 0
        for i in xrange(1000000000):
            tmp += i
        print 'process done'

if __name__ == '__main__':

    t = Test()
    pools = multiprocessing.Pool()
    for i in xrange(4):
        pools.apply_async(t.test)
    pools.close()
    pools.join()

this result is nothing, this pools don't call t.test! I can't understand what happended. Why is this?

I think you are running into [this](http://stackoverflow.com/a/1816969/377366) problem. Bound methods on objects basically don't work with multiprocessing. The solution depends on your situation. Which part of your real code is heavy? With multiprocessing you want to minimize the amount of data that is communicated between processes. — KobeJohn, Sep 01 '15 at 14:59
However your code does work (with minor modifications) in py3. I think py3 now allows pickling bound methods. — KobeJohn, Sep 01 '15 at 15:16

taesu · Answer 1 · 2015-09-02T03:38:42.497

instead of using pool, you can simply collect the jobs in a list:

import multiprocessing

class Test(multiprocessing.Process):
    def run(self):
        print 'processing.....'
        tmp = 0
        for i in xrange(10):
            tmp += i
        print 'process done'
        return 1

if __name__ == '__main__':
    jobs = []
    for i in range(5):
        t = Test()
        jobs.append(t)
        t.start()

the list jobs will be able to tell you if the process has finished or not, ultimately giving you the same effect as using pool.

if you wanna make sure that all jobs are done:

if __name__ == '__main__':
    jobs = []
    for i in range(5):
        t = Test()
        jobs.append(t)
        t.start()
    not_done = any(job.is_alive() for job in jobs)
    while not_done:
        not_done = any(job.is_alive() for job in jobs)
    print 'job all done'

How can I know all jobs if the process has finished? I want when all jobs is done, print a warn( "all jobs done"). — xina1i, Sep 02 '15 at 02:29
see edit. processes have built in func called `is_alive() `, which returns `false` if the job is done — taesu, Sep 02 '15 at 03:39

Multiprocessing works with standalone method but not with class instance method

1 Answers1