How to specify a certain keyword argument in Python multiprocessing

Question

For a function with many arguments, including position and keyword ones. How to specify a list to a keyword argument without repeat previous keyword arguments?

E.g.

import multiprocessing
from functools import partial

def foo(a, b, c=1, d=2, e=3):
    print(multiprocessing.current_process().name)
    print(a + b + c + d + e)
    return a + b + c + d + e

with multiprocessing.Pool(processes=2) as pool:
    # I would like to give [1,2,3,4] to "e", something like
    # dic = {'e': [1,2,3,4]}
    # Yet we still have c = 1, d = 2.
    results = pool.map(partial(foo, 2, 3), dic)

The argument list can be long, and I don't want to type "c=1, d=2" in every partial function.

I know it's possible to use multiprocessing.apply, but I would like keep consistent with other usages of this function, and using map only.

Is it always the same keyword (i.e. can the `'e'` be hard-coded) or could that vary? — alani, Sep 05 '20 at 04:31
No, it cannot be hardcoded, which makes it somewhat difficult for me.. — hzfmer, Sep 06 '20 at 05:27

alani · Accepted Answer · 2020-09-07T05:56:16.763

I think you are going to need some kind of wrapper function because it seems that filling keyword arguments with partial can't be done. At the most general, you could have a caller function that takes a tuple of (function, args, kwargs) -- and pass to pool.map this generic caller and a list of such 3-element tuples:

import multiprocessing
from functools import partial

def foo(a, b, c=1, d=2, e=3):
    print(multiprocessing.current_process().name)
    print(a, b, c, d, e, a + b + c + d + e)
    return a + b + c + d + e

def generic_caller(tup):
    func, args, kwargs = tup
    return func(*args, **kwargs)

with multiprocessing.Pool(processes=2) as pool:

    tups = [(foo, [2, 3], {'e': val})
            for val in [1, 2, 3, 4]]

    results = pool.map(generic_caller, tups)
    print(results)

This gives the following (note that I added the individual variables to the print statement inside foo):

ForkPoolWorker-2
2 3 1 2 1 9
ForkPoolWorker-2
2 3 1 2 2 10
ForkPoolWorker-2
2 3 1 2 4 12
ForkPoolWorker-1
2 3 1 2 3 11
[9, 10, 11, 12]

Or you might prefer to hard-code func and/or args in the wrapper, and omit any hard-coded items from the tuples that are passed.

Unfortunately solutions involving using a closure as the wrapper function (to avoid having to pass the same values of a and b every time) run into problems that this is not picklable, so would work with map but not pool.map.

Accpeted. Thank you! I guess in `generic_caller`, you mean return **func(*args, **kwargs)** though.. — hzfmer, Sep 07 '20 at 05:53
Glad it worked. I must admit, I started by trying to write a closure to solve this, and then came up against the not pickable problem - hence the last paragraph. — alani, Sep 07 '20 at 05:55

How to specify a certain keyword argument in Python multiprocessing

1 Answers1