sequential processing faster than pool processing

Question

I was trying to understand the use of map with multiprocessing. I have written the following python program for that purpose. But the results seems to confuse me.

from multiprocessing import Pool                                                                                                                                                                             
import time

def f(x):
    return x*x 

if __name__ == '__main__':
    p = Pool(5)
    l = [x for x in range(2000000)]
    start = time.clock()
    p.map(f, l)
    end = time.clock()
    print('pool processing time {}'.format(end - start))
    start = time.clock()
    map(f, l)
    end = time.clock()
    print('sequential processing time {}'.format(end - start))

The output that I get is given below.

  pool processing time 5.576627
  sequential processing time 3.220387

Why is the sequential processing time greater than pool processing time? I am running this code on a Linux (Ubuntu 14.04 VM) that has two CPU's allocated to it.

Possible duplicate of [multiprocessing.Pool() slower than just using ordinary functions](http://stackoverflow.com/questions/20727375/multiprocessing-pool-slower-than-just-using-ordinary-functions) — pppery, Nov 08 '15 at 20:12

score 3 · Accepted Answer · answered Nov 08 '15 at 20:14

Your function f is too simple to benefit here.

The way multiprocessing works is to spin off copies of your entire Python program. If your system has multiple CPUs, these copies can then run in parallel on the separate CPUs. You can think of them as being in a sort of master/slave arrangement, with the original Python program being the master on CPU A, and some slaves—5, in this case—on CPUs B through F. (They don't have to have this sort of master/slave relationship but I think most people find it easier to think about this way.)

Then, every time your master asks for some slave-side computation, the master packages up the argument values (using pickle) and sends them to the slave.¹ The slave does the requested computation, pickles the answer, and ships it back to the master.

In your case the argument is each value in the list (plus the function to call, see footnote) and the result is the square of the value in the list. It takes significantly longer to pack and unpack the values than it does to do the computation, so this is a net loss.

If you did more computation (as compared to basic communications overhead) you would likely see a net win, though if there are only two CPUs, making a large pool will be counterproductive.

¹The function-to-call is also wrapped up via pickle.

I have changed my function from a `square` to a `power of 4` and multiprocessing became faster than sequential processing. — liv2hak, Nov 08 '15 at 20:22

sequential processing faster than pool processing

1 Answers1