Multiprocessing with numpy makes Python quit unexpectedly on OSX

Question

I've run into a problem, where Python quits unexpectedly, when running multiprocessing with numpy. I've isolated the problem, so that I can now confirm that the multiprocessing works perfect when running the code stated below:

import numpy as np
from multiprocessing import Pool, Process
import time
import cPickle as p

def test(args):
    x,i = args
    if i == 2:
        time.sleep(4)
    arr = np.dot(x.T,x)
    print i

if __name__ == '__main__':
    x = np.random.random(size=((2000,500)))
    evaluations = [(x,i) for i in range(5)]
    p = Pool()
    p.map_async(test,evaluations)
    p.close()
    p.join()

The problem occurs when I try to evaluate the code below. This makes Python quit unexpectedly:

import numpy as np
from multiprocessing import Pool, Process
import time
import cPickle as p

def test(args):
    x,i = args
    if i == 2:
        time.sleep(4)
    arr = np.dot(x.T,x)
    print i

if __name__ == '__main__':
    x = np.random.random(size=((2000,500)))
    test((x,4)) # Added code
    evaluations = [(x,i) for i in range(5)]
    p = Pool()
    p.map_async(test,evaluations)
    p.close()
    p.join()

Please help someone. I'm open to all suggestions. Thanks. Note: I have tried two different machines and the same problem occurs.

I ran your code on Windows7/64bit using WinPython. Both cases executed and exited without errors. — alandarev, Oct 31 '13 at 11:29
Sorry for that. Interesting that it works on Windows. Any Apple users who can explain to me why this occurs? — Maal, Oct 31 '13 at 11:36
I might have misleaded you, when making my rage-comment about Apple using "unexpected" term everywhere. I am hightly doubting your issue is OS specific. Could you try running the script you posted against clean Python with numpy install to see if issue persists? — alandarev, Oct 31 '13 at 11:42
I've just tried on a third apple workstation with clean Python and Numpy installed. It is the same behaviour unfortunately. — Maal, Oct 31 '13 at 11:53
@Maal This script works for me on OS X 10.6.8 using the Anaconda Python Distribution v1.7 (Python 2.7.5 and Numpy 1.7.1) — JoshAdel, Oct 31 '13 at 17:25
@JoshAdel You're probably using a version of Anaconda which links to the MKL library, not Apple's Accelerate (which is where the problem lies, as in the links from Joe's answer). You can check with `otool -L $(python -c 'from numpy.core import _dotblas; print(_dotblas.__file__)')`. — Danica, Oct 31 '13 at 19:55
I had the same problem. What I did was to install numpy/scipy using OpenBLAS instead. There's a nice explanation here: http://stackoverflow.com/a/14391693/777706 — Oriol Nieto, Aug 20 '14 at 10:57

score 6 · Answer 1 · edited May 23 '17 at 12:25

6

This is a known issue with multiprocessing and numpy on MacOS X, and a bit of a duplicate of:

segfault using numpy's lapack_lite with multiprocessing on osx, not linux

http://mail.scipy.org/pipermail/numpy-discussion/2012-August/063589.html

The answer seems to be to use a different BLAS other than the Apple accelerate framework when linking Numpy... unfortunate :(

edited May 23 '17 at 12:25

Community

1
1

answered Oct 31 '13 at 19:44

Joe

227
1
2

2

As @ogrisel noted in a comment on the first link there, there's also a `forkserver` mode in new versions of multiprocessing that can work around the problem. – Danica Oct 31 '13 at 19:54

score 6 · Accepted Answer · answered Nov 20 '13 at 12:43

I figured out a workaround to the problem. The problem occurs when Numpy is used together with BLAS before initializing a multiprocessing instance. My workaround is simply to put the Numpy code (running BLAS) into a single process and then running the multiprocessing instances afterwards. This is not a good coding style, but it works. See example below:

Following will fail - Python will quit:

import numpy as np
from multiprocessing import Pool, Process

def test(x):
    arr = np.dot(x.T,x) # On large matrices, this calc will use BLAS.

if __name__ == '__main__':
    x = np.random.random(size=((2000,500))) # Random matrix
    test(x)
    evaluations = [x for _ in range(5)]
    p = Pool()
    p.map_async(test,evaluations) # This is where Python will quit, because of the prior use of BLAS.
    p.close()
    p.join()

Following will succeed:

import numpy as np
from multiprocessing import Pool, Process

def test(x):
    arr = np.dot(x.T,x) # On large matrices, this calc will use BLAS.

if __name__ == '__main__':
    x = np.random.random(size=((2000,500))) # Random matrix
    p = Process(target = test,args = (x,))
    p.start()
    p.join()
    evaluations = [x for _ in range(5)]
    p = Pool()
    p.map_async(test,evaluations)
    p.close()
    p.join()

Multiprocessing with numpy makes Python quit unexpectedly on OSX

2 Answers2

Linked