Python Multiprocessing - How to pass kwargs to function?

Question

How do I pass a dictionary to a function with Python's Multiprocessing? The Documentation: https://docs.python.org/3.4/library/multiprocessing.html#reference says to pass a dictionary, but I keep getting

TypeError: fp() got multiple values for argument 'what'

Here's the code:

from multiprocessing import Process, Manager
   
def fp(name, numList=None, what='no'):
        print ('hello %s %s'% (name, what))
        numList.append(name+'44')
 
if __name__ == '__main__':

    manager = Manager()
 
    numList = manager.list()
    for i in range(10):
        keywords = {'what':'yes'}
        p = Process(target=fp, args=('bob'+str(i)), kwargs={'what':'yes'})
        p.start()
        print("Start done")
        p.join()
        print("Join done")
    print (numList)

I believe that [this thread](http://stackoverflow.com/questions/6832554/python-multiprocessing-how-do-i-share-a-dict-among-multiple-processes) will help answer your question. — Nathan Marx, Aug 12 '16 at 02:03
Also here: https://stackoverflow.com/questions/34031681/passing-kwargs-with-multiprocessing-pool-map — Brian Larsen, Aug 04 '17 at 19:22

score 31 · Accepted Answer · edited Aug 07 '20 at 07:07

When I ran your code, I got a different error:

TypeError: fp() takes at most 3 arguments (5 given)

I debugged by printing args and kwargs and changing the method to fp(*args, **kwargs) and noticed that "bob_" was being passed in as an array of letters. It seems that the parentheses used for args were operational and not actually giving you a tuple. Changing it to the list, then also passing in numList as a keyword argument, made the code work for me.

from multiprocessing import Process, Manager

def fp(name, numList=None, what='no'):
    print ('hello %s %s' % (name, what))
    numList.append(name+'44')

if __name__ == '__main__':

    manager = Manager()

    numList = manager.list()
    for i in range(10):
        keywords = {'what': 'yes', 'numList': numList}
        p = Process(target=fp, args=['bob'+str(i)], kwargs=keywords)
        p.start()
        print("Start done")
        p.join()
        print("Join done")
    print (numList)

Thanks! I didn't think it was possible and ended up coding it a different way. But I'll use this in the future! Also adding a comma after 'bob'+str(i) makes it a list 'bob'+str(i), — Jim Factor, Aug 12 '16 at 18:27

Python Multiprocessing - How to pass kwargs to function?

1 Answers1

Linked