0

I want to create a pool over a function of several variables where one is held fixed. If I was using a basic map I would have something like: 

def test_map(a,b):
    return a+b
serial_map = map(lambda x: test_map(2,x), [1, 1, 3])

But this does not work for a pool map: 


from multiprocessing import Pool
p = Pool(4)
p.map(lambda x: test_map(2,x), [1, 1, 3])

I'm aware this causes a pickling error. My point is that I would like a workaround if it is possible. I know for regular maps the idiomatic way of handling this issue is with lambdas, I would like to know the pool.map way of handling this problem.

Daniel Marchand
  • 584
  • 8
  • 26
  • 1
    https://stackoverflow.com/questions/8804830/python-multiprocessing-picklingerror-cant-pickle-type-function – RomanPerekhrest Nov 18 '19 at 14:00
  • 1
    ***"pool.map way of handling this problem."***: `p.map(test_map, ((2,x) for x in [1, 1, 3]))` – stovfl Nov 18 '19 at 14:13
  • stovfl I will vote reopen the question. Please write this as the answer and I will accept. @Mad Physicist this question is *not* a duplicate as the answer needn't involve any pickling in the end as stovfl's answer has demonstrated. – Daniel Marchand Nov 18 '19 at 14:19
  • 1
    @DanielMarchand. Agreed. I closed it because the answer to the other question would work, but this is more elegant. Keep in mind that this *does* involve pickling, and the fact that your function is top-level is the *only* reason this works. – Mad Physicist Nov 18 '19 at 14:42
  • 1
    Did you try `from functools import partial` and `p.map(partial(test_map, a=2), [1, 1, 3])`? – Guimoute Nov 18 '19 at 15:58

1 Answers1

0

In this case it makes more sense to use the starmap function. 

from multiprocessing import Pool

def test_map(a,b):
    return a+b
p = Pool(4)
p.starmap(test_map, [(2,x) for x in [1, 1, 3]])
Daniel Marchand
  • 584
  • 8
  • 26