3

I am trying to get to grips with multiprocessing in Python. I started by creating this code. It simply computes cos(i) for integers i and measures the time taken when one uses multiprocessing and when one does not. I am not observing any time difference. Here is my code:

    import multiprocessing
    from multiprocessing import Pool
    import numpy as np
    import time


    def tester(num):
        return np.cos(num)


    if __name__ == '__main__':


        starttime1 = time.time()
        pool_size = multiprocessing.cpu_count()
        pool = multiprocessing.Pool(processes=pool_size,
                            )
        pool_outputs = pool.map(tester, range(5000000))
        pool.close()
        pool.join()
        endtime1 = time.time()    
        timetaken = endtime1 - starttime1  

        starttime2 = time.time()
        for i in range(5000000):
            tester(i)
        endtime2 = time.time()
        timetaken2 = timetaken = endtime2 - starttime2

        print( 'The time taken with multiple processes:', timetaken)
        print( 'The time taken the usual way:', timetaken2)

I am observing no (or very minimal) difference between the two times measured. I am using a machine with 8 cores, so this is surprising. What have I done incorrectly in my code?

Note that I learned all of this from this. http://pymotw.com/2/multiprocessing/communication.html

I understand that "joblib" might be more convenient for an example like this, but the ultimate thing that this needs to be applied to does not work with "joblib".

shilovk
  • 11,718
  • 17
  • 75
  • 74
Kyle Powell
  • 31
  • 2
  • 3
  • https://wiki.python.org/moin/GlobalInterpreterLock – Sam Jul 24 '15 at 10:43
  • 1
    Dear sir, Thank you. I read about this GIL before. I thought that the use of multiprocessing (as I have done here, I believe) gets around this issue. I had originally intended to use OpenMP, but found that this is not possible in Python due to the GIL. I guess that Python is just not built for high-speed, parallel tasks. – Kyle Powell Jul 24 '15 at 10:45
  • thanks for the heads up, it looks like you're right. From the docs, `multiprocessing` "effectively side-step[s] the Global Interpreter Lock by using subprocesses instead of threads". Makes sense. So disregard my earlier comment. – Sam Jul 24 '15 at 10:46

3 Answers3

8

Your job seems the computation of a single cos value. This is going to be basically unnoticeable compared to the time of communicating with the slave.

Try making 5 computations of 1000000 cos values and you should see them going in parallel.

6502
  • 112,025
  • 15
  • 165
  • 265
4

First, you wrote :

timetaken2 = timetaken = endtime2 - starttime2

So it is normal if you have the same times displayed. But this is not the important part.

I ran your code on my computer (i7, 4 cores), and I get :

('The time taken with multiple processes:', 14.95710802078247)
('The time taken the usual way:', 6.465447902679443)

The multiprocessed loop is slower than doing the for loop. Why?

The multiprocessing module can use multiple processes, but still has to work with the Python Global Interpreter Lock, wich means you can't share memory between your processes. So when you try to launch a Pool, you need to copy useful variables, process your calculation, and retrieve the result. This cost you a little time for every process, and makes you less effective.

But this happens because you do a very small computation : multiprocessing is only useful for larger calculation, when the memory copying and results retrieving is cheaper (in time) than the calculation.

I tried with following tester, which is much more expensive, on 2000 runs:

def expenser_tester(num):
    A=np.random.rand(10*num) # creation of a random Array 1D
    for k in range(0,len(A)-1): # some useless but costly operation
        A[k+1]=A[k]*A[k+1] 
    return A

('The time taken with multiple processes:', 4.030329942703247)
('The time taken the usual way:', 8.180987119674683)

You can see that on an expensive calculation, it is more efficient with the multiprocessing, even if you don't always have what you could expect (I could have a x4 speedup, but I only got x2) Keep in mind that Pool has to duplicate every bit of memory used in calculation, so it may be memory expensive.

If you really want to improve a small calculation like your example, make it big by grouping and sending a list of variable to the pool instead of one variable by process.

You should also know that numpy and scipy have a lot of expensive function written in C/Fortran and already parallelized, so you can't do anything much to speed them.

CoMartel
  • 3,521
  • 4
  • 25
  • 48
  • 1
    Thank you very much. Firstly, when I fixed that mistake, I found a difference in times taken. I now understand that using multiple processes should only be used with an expensive calculation. Are numpy and scipy operations automatically parallelized? So, if I apply a numpy function to an array or make a calculation with a large array, will it automatically be parallelized? – Kyle Powell Jul 24 '15 at 17:38
  • It depends on the function you are using, not all are parallelized, but most of numpy's array manipulations are parallelized. You should see the documentation for the functions you are interested to check if they are parallelized. If the function is parrallelized, you don't have anything to do to improve it (in Python anyway) – CoMartel Jul 26 '15 at 18:07
0

If the problem is cpu bounded then you should see the required speed-up (if the operation is long enough and overhead is not significant). But when multiprocessing (because memory is not shared between processes) it's easier to have a memory bound problem.

Community
  • 1
  • 1
Marc Cayuela
  • 1,504
  • 13
  • 26