python code with mulitprocessing only spawns one process each time

Question

I have a simple python code, trying to convert a batch of jpg file to png. But this code only spawn one subprocess each time on my linux machine, what's the problem with this code?

from scipy.misc import imread, imsave
from multiprocessing import Process, util
import logging

util.log_to_stderr(level=logging.DEBUG)
def imgcvt(inf, outf):
    im=imread(inf)
    imsave(outf,im)

for i in range(3):
    run_list = [];
    threads = [];
    for j in range(1,21):
        fn = str(i*20+j+1)
        run_list.append((fn+".jpg", fn+".png"))
    for dev in run_list:
        proc = Process(target=imgcvt, args=dev)
        proc.start()
        proc.join()

thanks

Answer 1

Doing a proc.join() after starting every process means you are waiting for it to terminate before continuing.

Instead, you should store all Process objects to a list, and then join on all of them:

processes = []
for dev in run_list:
    proc = Process(target=imgcvt, args=dev)
    proc.start()
    processes.append(proc)
for proc in processes:
    proc.join()

Answer 2

Consider using a multiprocessing.Pool object. It starts a few processes, then converts your images in parallel until they're all done. It's nice because it's easy to control, and returns information from every function back to the original caller.

source

import multiprocessing, time

def convert( (inpath,outpath) ):
    if 1:
        time.sleep(1)
        print 'convert:',inpath,outpath
    else:
        im = imread(inpath)
        imsave(outpath, im)
    return outpath

if __name__=='__main__':
    run_list = []
    for j in range(5):
        fn = str(j+1)
        run_list.append( (fn+".jpg", fn+".png") )

    pool = multiprocessing.Pool()   # one Process per CPU
    print pool.map(convert, run_list)
    pool.close()
    pool.join()                 # wait for all jobs to finish

output

convert: 3.jpg 3.png
convert: 2.jpg 2.png
convert: 4.jpg 4.png
convert: 1.jpg 1.png
convert: 5.jpg 5.png
['1.png', '2.png', '3.png', '4.png', '5.png']