Multiprocessing incompatible with NumPy

Question

I am trying to run a simple test using multiprocessing. The test works well until I import numpy (even though it is not used in the program). Here is the code:

from multiprocessing import Pool
import time
import numpy as np #this is the problematic line


def CostlyFunc(N):
    """"""
    tstart = time.time()
    x = 0
    for i in xrange(N):
        for j in xrange(N):
            if i % 2: x += 2
            else: x -= 2       
    print "CostlyFunc : elapsed time %f s" % (time.time() - tstart)
    return x

#serial application
ResultList0 = []
StartTime = time.time()
for i in xrange(3):
    ResultList0.append(CostlyFunc(5000))
print "Elapsed time (serial) : ", time.time() - StartTime


#multiprocessing application
StartTime = time.time()
pool = Pool()
asyncResult = pool.map_async(CostlyFunc, [5000, 5000, 5000])
ResultList1 = asyncResult.get()
print "Elapsed time (multiporcessing) : ", time.time() - StartTime

If I don't import numpy the result is:

CostlyFunc : elapsed time 2.866265 s
CostlyFunc : elapsed time 2.793213 s
CostlyFunc : elapsed time 2.794936 s
Elapsed time (serial) :  8.45455098152
CostlyFunc : elapsed time 2.889815 s
CostlyFunc : elapsed time 2.891556 s
CostlyFunc : elapsed time 2.898898 s
Elapsed time (multiporcessing) :  2.91595196724

The total elapsed time is similar to the time required for 1 process, meaning that the computation has been parallelized. If I do import numpy the result becomes :

CostlyFunc : elapsed time 2.877116 s
CostlyFunc : elapsed time 2.866778 s
CostlyFunc : elapsed time 2.860894 s
Elapsed time (serial) :  8.60492110252
CostlyFunc : elapsed time 8.450145 s
CostlyFunc : elapsed time 8.473006 s
CostlyFunc : elapsed time 8.506402 s
Elapsed time (multiporcessing) :  8.55398178101

The total time elapsed is the same for both serial and multiprocessing methods because only one core is used. It is clear that the problem comes from numpy. Is it possible that I have an incompatibility between my versions of multiprocessing and NumPy?

I am currently using Python2.7, NumPy 1.6.2 and multiprocessing 0.70a1 on linux

That is very strange- It appears to work fine on OSX with Python 2.7 and NumPy 1.7. From the timings it looks like three cores are used, but the processing time is slowed- can you confirm this? — Daniel, Aug 07 '13 at 14:33
Thanks for you answer. I am quite sure the computation is made on only 1 core when I import NumPy (I checked with mpstat (http://linuxcommand.org/man_pages/mpstat1.html)). It appears that the same core computes the 3 jobs at the same time : so each jobs takes ~8.5 sec but the total time is also ~8.5 sec. — user2660966, Aug 07 '13 at 14:48
I tried with numpy 1.6.1, numpy 1.6.2 and numpy 1.7.1 ... same problem — user2660966, Aug 09 '13 at 13:14
Numpy can mess with core affinity on import, such that all child processes end up vying for the same core. Take a look at [my question/answer here](http://stackoverflow.com/questions/15639779/what-determines-whether-different-python-processes-are-assigned-to-the-same-or-d) for a workaround. — ali_m, Aug 13 '13 at 11:21

score 4 · Answer 1 · edited May 23 '17 at 12:22

(First Post sorry if it is not well formulated or alligned)

You can stop Numpy to use multithreading by seting the MKL_NUM_THREADS to 1

Under debian I used:

export MKL_NUM_THREADS=1

Source from related stackoverflow post: Python: How do you stop numpy from multithreading?

Result:

user@pc:~/tmp$ python multi.py
CostlyFunc : elapsed time 3.847009 s
CostlyFunc : elapsed time 3.253226 s
CostlyFunc : elapsed time 3.415734 s
Elapsed time (serial) :  10.5163660049
CostlyFunc : elapsed time 4.218424 s
CostlyFunc : elapsed time 5.252429 s
CostlyFunc : elapsed time 4.862513 s
Elapsed time (multiporcessing) :  9.11713695526

user@pc:~/tmp$ export MKL_NUM_THREADS=1

user@pc:~/tmp$ python multi.py
CostlyFunc : elapsed time 3.014677 s
CostlyFunc : elapsed time 3.102548 s
CostlyFunc : elapsed time 3.060915 s
Elapsed time (serial) :  9.17840886116
CostlyFunc : elapsed time 3.720322 s
CostlyFunc : elapsed time 3.950583 s
CostlyFunc : elapsed time 3.656165 s
Elapsed time (multiporcessing) :  7.399310112

I am not sure if that helps because I guess eventually you want numpy to run in parallel maybe try to adjust the number of threads for numpy to your machine.

To the person who downvoted: if you downvote - please explain why and ask questions in the comment section. — Jon, Dec 13 '13 at 11:34

score 1 · Answer 2 · edited May 23 '17 at 11:43

From the comments on your question, have a look at that link @Ophion no, but I have flagged it as a duplicate of Why does multiprocessing use only a single core after I import numpy? – ali_m Aug 22 at 9:06

I would check to see if you are using an optimized version of BLAS. I have found that some generic installs of numpy do not deliver and optimized version of this lib. From my install you can note that is points to libf77blas.so, libcblas.so, libatlas.so.

Here are the instructions to build an optimized version of BLAS: http://docs.scipy.org/doc/numpy/user/install.html

From with in python:

import numpy.core._dotblas

>>> numpy.core._dotblas.__file__

## output:

'PYTHONHOME/lib/python2.7/site-packages/numpy/core/_dotblas.so'

From your terminal:

$ ldd 'PYTHONHOME/lib/python2.7/site-packages/numpy/core/_dotblas.so'
linux-vdso.so.1 =>  (0x00007fff241ff000)
libf77blas.so => /opt/arch/intel/lib/libf77blas.so (0x00007f6050647000)
libcblas.so => /opt/arch/intel/lib/libcblas.so (0x00007f6050429000)
libatlas.so => /opt/arch/intel/lib/libatlas.so (0x00007f604fbf1000)
libpython2.7.so.1.0 => 'PYTHONHOME/lib/libpython2.7.so.1.0 (0x00007f604f817000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f604f5f9000)
libc.so.6 => /lib64/libc.so.6 (0x00007f604f266000)
libgfortran.so.3 => /usr/lib64/libgfortran.so.3 (0x00007f604ef74000)
libm.so.6 => /lib64/libm.so.6 (0x00007f604ecef000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f604eaeb000)
libutil.so.1 => /lib64/libutil.so.1 (0x00007f604e8e8000)

/lib64/ld-linux-x86-64.so.2 (0x0000003c75e00000)

Multiprocessing incompatible with NumPy

2 Answers2

Linked