7

This question may be a duplicate. However, I read lot of stuff around on this topic, and I didn't find one that matches my case - or at least, I didn't understood it.

Sorry for the inconvenance.

What I'm trying to do is fairly common, passing a list of kwargs to pool.starmap(), to achieve multiprocessing.

Here's my reduced case:

def compute(firstArg, **kwargs): # A function that does things
    # Fancy computing stuff...
    # Saving to disk...
    return True

if __name__ ==  '__main__':
    args = [{
        'firstArg': "some data",
        'otherArg': "some other data"
        'andAnotherArg': x*2
    } for x in range(42)]

    pool = Pool(4)
    pool.starmap(compute, args)
    pool.close()
    pool.terminate()

I supposed starmap() will unpack the dict and pass it to compute() as keyword args, but looking at the source code (see also l.46), it sends only keys (or values ?).

So it raises :

TypeError: compute() takes 1 positional argument but 3 were given

It must be a clear, straight forward way to do this... Any help would be appreciated.

Here's a quite similar question : Python Multiprocessing - How to pass kwargs to function?

Community
  • 1
  • 1
Le Minaw
  • 835
  • 7
  • 17

1 Answers1

4

You could use a tiny proxy function:

def proxy(Dict):
    return compute(**Dict)

pool.map(proxy, args)

Or, since you don't need the proxy function polluting the namespace:

pool.map(lambda Dict: compute(**Dict), args)
ForceBru
  • 43,482
  • 10
  • 63
  • 98
  • 1
    Great ! Just used it in a lambda, that's perfect. Thanks for your help ! – Le Minaw May 13 '17 at 21:03
  • Is there anyway to unpack list of dict's? I need to pass the whole dict, not just kwargs. Thanks! – New2coding Jan 11 '19 at 16:34
  • I have a function like this: def proc(arg): os.system(arg['args'] % (arg['exe'], arg['env'], arg['taskName'], arg['searchPath'])) and would like to pass list of dicts within pool thought. – New2coding Jan 11 '19 at 16:50
  • @New2Python, just don't unpack the individual dictionaries: `pool.map(proc, args)`. – ForceBru Jan 11 '19 at 16:54