Passing a list of items to multiprocessing function

Question

I have the following function:

def create_job(target, args):
    p = multiprocessing.Process(target=target, args=args)
    p.start()
    return p

The above function gets called by:

ps = create_job(mc.collect_categories, (cat_ids),)

Being cat_ids a list of 'ids': ['ML68', 'MA453','MA33']

When I run the code I get the following error(with a list of 10 cat_ids):

TypeError: collect_categories() takes exactly 2 arguments (11 given)

I tried to pass the values with the '*' to unpack it but that wasn't the problem.

score 8 · Accepted Answer · edited May 23 '17 at 10:31

(cat_ids) is identical to cat_ids. Instead use (cat_ids, ) to denote a tuple with one element:

ps = create_job(mc.collect_categories, (cat_ids, ),)

Alternatively, you could define create_job using *args:

def create_job(target, *args):
    p = multiprocessing.Process(target=target, args=args)
    p.start()
    return p

and call create_job with:

ps = create_job(mc.collect_categories, cat_ids)

Inside create_job, the variable args will be a tuple of all the positional arguments (except the first) passed to create_job -- exactly what you want to pass to Process.

Passing a list of items to multiprocessing function

1 Answers1