Python - using multiprocessing with multiple args doesn't work

Question

I'm trying to use multiprocessing with multiple args to print dummy values, but this doesn't seems to work. I get error

"f() missing 2 required positional arguments:..."

for the following code:

from multiprocessing import Pool

class tryProcessing:
    def f(self, arr, arg1, arg2):
        print(arr + " " + arg1 + " " + arg2)

    def func(self, arr, arg1, arg2):
        arg1 = "hi"
        arg2 = "hello"
        arr_a = ['1','2']
        arr_b = ['3','4','5']
        p = Pool(Processes=2)
        p.map(self.f, [[a, arg1, arg2], [b, arg1, arg2]])
        p.close

What am I doing wrong?

P.s. in this answer, he did something similar, I don't understand why his works, and mine doesn't.

Do the arguments need to be in tuples instead? `[(a, arg1, arg2), (b, arg1, arg2)]` — mattrea6, Oct 16 '19 at 12:18

score 1 · Answer 1 · answered Oct 16 '19 at 12:21

1

You pass one arguments, and it's an entire list.


test = map(lambda x : x, [[a, arg1, arg2], [b, arg1, arg2]]) 
print(next(test))

You can update you f func like this.


def f(self, *args):
    arr, arg1, arg2 = args
    print(f"{arr} {arg1} {arg2}")

answered Oct 16 '19 at 12:21

Florian Bernard

2,561
1
9
22

score 1 · Accepted Answer · answered Oct 16 '19 at 12:26

You are looking for starmap, which expected the iterable to contain nested iterables of arguments to be expanded as function arguments. It uses star (splat) notation to expand, hence the name.

P.S. you never actually call p.close at the end of your function.

score 0 · Answer 3 · answered Oct 16 '19 at 12:43

There are a couples of difference with your approach compared with the solution you posted it's link there.

Notice positional arguments def someaction(a, b=1, c=2, d=3)
And you don't have positional argument def f(self, arr, arg1, arg2)

That might explain the error you are getting above. Tweaking your code this works

from multiprocessing import Pool

class tryProcessing:
    def f(self, arr):
        print(arr)

    def func(self, arr, arg1, arg2):
        arg1 = "hi"
        arg2 = "hello"
        arr_a = ['1','2']
        arr_b = ['3','4','5']
        p = Pool(2)

        data = [["a1", "b1", "c1", "d1"],
        ["a2", "b2", "c2", "d2"],
        ["a3", "b3", "c3", "d3"], ]

        p.map( self.f, data)
        p.close

t = tryProcessing()
t.func("adfasf", "dfaf", "daf")

Python - using multiprocessing with multiple args doesn't work

3 Answers3