How to apply keyword-only arguments to a function in a multiprocessing pool?

Question

I have a function that takes a keyword-only argument and want to run it in a process pool. How do I pass my entries from an iterable to the function in the process as a keyword argument?

import multiprocessing

greetees = ('Foo', 'Bar')

def greet(*, greetee):
    return f'Hello, {greetee}!'

I tried using multiprocessing.map:

greetings = multiprocessing.Pool(2).map(greet, greetees)
for greeting in greetings:
    print(greeting)

But that raises an exception, as expected:

multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
TypeError: greet() takes 0 positional arguments but 1 was given
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/bengt/Projekte/gitlab.com/PFASDR/PFASDR.Code.Main/pfasdr/neural/multi_pool_kwargs.py", line 10, in <module>
    greetings = multiprocessing.Pool(2).map(greet, greetees)
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 266, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 644, in get
    raise self._value
TypeError: greet() takes 0 positional arguments but 1 was given

It works fine if I remove the asterisk to not require the arguments to be keyword-only:

[...]
def greet(greetee):
    return f'Hello, {greetee}!'
[...]

Output:

Hello, Foo!
Hello, Bar!

Bengt · Answer 1 · 2019-10-18T15:20:11.683

1

A solution here is to use Pool.apply or Pool.apply_async:

greetings = list(
    multiprocessing.Pool(2).apply(greet, kwds={'greetee': greetees[i]})
    for i in range(len(greetees))
)
for greeting in greetings:
    print(greeting)

Output:

Hello, Foo!
Hello, Bar!

edited Oct 18 '19 at 15:20

answered Oct 18 '19 at 15:14

Bengt

14,011
7
48
66

Should this be the accepted answer or is there a better way? – Bengt Oct 18 '19 at 15:16
Thanks, I did see this answer, but did not understand that it was concerned with the same problem. Yes, that does actually work. I will post another answer. – Bengt Oct 18 '19 at 15:33
You should post in the other question. Do you agree that yours is essentially a duplicate? – Mad Physicist Oct 18 '19 at 15:34
1

No, I do not think these are duplicates. They try and pass 3 static keyword arguments to the function (`b`, `c`, `d`) using `functools.partial` and one (`a`) as a positional argument using `Pool.map`. I cannot do the latter, because my function's argument is keyword-only. Hence, I can only pass the keyword-only argument using `apply` or `apply_async`. – Bengt Oct 18 '19 at 15:54

Bengt · Answer 2 · 2019-10-18T16:07:22.297

Courtesy of Mad Physicist and this QnA, one can use functools.partial to inject keyword-only arguments into a function:

from functools import partial
greetings = []
for i in range(len(greetees)):
    kwargs = {'greetee': greetees[i]}
    greet_partial = partial(greet, **kwargs)
    greetings.append(multiprocessing.Pool(2).apply(greet_partial))

Or with less variable bleed:

from functools import partial
greetings = [
    multiprocessing.Pool(2).apply(
        partial(greet, **{'greetee': greetees[i]})
    )
    for i in range(len(greetees))
]

How to apply keyword-only arguments to a function in a multiprocessing pool?

2 Answers2