1

I am attempting to speed up some python code can only run single threaded. I an running many of these in a for loop and would like to parallelize it and save the results in a dictionary.

I've searched stack overflow and read the multiprocessing documentation but can't find a good solution.

Example of not parallelized:

%%time
# This only uses one thread! It's slow
mydict = {}
for i in range(20000000):
    mydict[i] = i**2

Returns:

CPU times: user 8.13 s, sys: 1.04 s, total: 9.17 s
Wall time: 9.21 s

and my dictionary is correct

print([mydict[i] for i in range(10)])
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

My attempt at parallelizing:

%%time
import multiprocessing as mp
from multiprocessing import Process, Manager
def square(d, i):
    d[i] = i**2

with mp.Manager() as manager:
    d = manager.dict()
    with manager.Pool(processes=4) as pool:
        pool.map(square, (d, range(20000000)))

Returns:

TypeError: square() missing 1 required positional argument: 'i'

Expected results are the correct dictionary but the time being roughly 1/4 of 9.21s.

Rob
  • 65
  • 8

1 Answers1

1

In case you have a target function with multiple parameters, you need pool.starmap(). .starmap() will unpack and map the argument-tuples in your iterable to the parameters of your target function. The iterable-parameter needs this layout for use with .starmap():

iterable = [(argA1, argB1), (argA2, argB2) ...]

Use itertools.repeat() to replicate the references for scalars like your d and create the iterable of argument-tuples with zip():

pool.starmap(square, zip(itertools.repeat(d), range(20)))
Darkonaut
  • 20,186
  • 7
  • 54
  • 65
  • Thanks! This worked. However in my example this ended up taking much longer to process than just a single thread. Hopefully it will improve the speed when I use it in the actual application. Very helpful to know abou the `starmap` function though! – Rob Jun 04 '19 at 19:44
  • 1
    @Rob You're welcome! Yeah, there's no guarantee that multiprocessing will always be faster because of the overhead involved. When you look at the pictures in chapter 8 of my answer [here](https://stackoverflow.com/a/54813527/9059420) you can get the idea how overhead (and chunking) can influence timings. – Darkonaut Jun 04 '19 at 19:49