Spawn parallel processes for function and pass several different arguments to function

Question

Hi everybody I took Jiaaro's solution as a template to convert it from threading to multiprocessing:

import multiprocessing
from function_repo import run
from time import time

vitems = ['02','63','25','0']

num_processes = (multiprocessing.cpu_count()/1)
threads = []

if __name__ == '__main__':
    begin = time()
    print begin
    # run until all the threads are done, and there is no data left
    while threads or vitems:
        if( len(threads) < (num_processes -1) ):

            p = multiprocessing.Process(target=run,args=[vitems.pop()])

            p.start()

            print p, p.is_alive()

            threads.append(p)

        else:

            for thread in threads:

                if not thread.is_alive():

                    threads.remove(thread)
    print  'Hello, finished'
    print  'Took: ',(time()-begin)

It works nicely, but now I want to pass a second argument that is not a list to the run function, like p = multiprocessing.Process(target=run,args=([vitems.pop()],second_arg))

This causes the function to break, because if I do so, the full list of vitems is passed to the function in every single process.

How can i still only pass one item of vitems per process to the function AND also pass a second non-list argument to the function run ?

thanks in advance and best regards!

score 0 · Accepted Answer · answered Jun 05 '15 at 09:52

p = multiprocessing.Process(target=run, args=[vitems.pop()]) does not pass a list to run, it passes the item poped from vitems as the first argument. args is supposed to be a list/tuple of arguments passed to run.

To pass multiple arguments just do:

p = multiprocessing.Process(target=run, args=(vitems.pop(), arg2, ...))

Spawn parallel processes for function and pass several different arguments to function

1 Answers1

Linked