2

I would like to improve the speed of this calculation by using the multiprocessing. But, I don't understand why my implementation (the second solution) doesn't work... Python "crash" (Python left in a unforeseen way - see below for a part of the report) without exception, or error

# -*- coding: utf-8 -*-
import time
import numpy as np
import multiprocessing as mp

a = 1.1
b = 0.1
d = 2

M = 1000
N = 500
P = 100

tab = np.random.random_sample([N,M])*1000
vectors = np.random.random_sample([P,M])*100
coef = np.random.random_sample([1,P])

def sol2(i):
    return np.dot(coef,(a + b*np.dot(tab,vectors.T).T)**d)


if __name__ == '__main__':
    #sol1
    start_time = time.time()
    for i in range(1000):
        res = np.dot(coef,(a + b*np.dot(tab,vectors.T).T)**d)
    print("--- %s seconds ---" % (time.time() - start_time))

    #sol2
    start_time = time.time()
    pool = mp.Pool()
    asyncResult = pool.map_async(sol2, xrange(1))# 1 to test
    res = asyncResult.get()
    print("--- %s seconds ---" % (time.time() - start_time))

Apple Report: Process: Python [96770] Path: /usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python Identifier: Python Version: 2.7.9 (2.7.9) Code Type: X86-64 (Native) Parent Process: Python [96766] Responsible: iTerm [96704] User ID: 501

Date/Time: 2015-05-24 23:00:03.942 +0200 OS Version: Mac OS X 10.10.3 (14D136) Report Version: 11 Anonymous UUID: AC6120B2-F5B6-D63F-641A-81A79258B4D8

Sleep/Wake UUID: 7AF8B0D6-5E2E-4794-84BE-0DABEE821CB9

Time Awake Since Boot: 150000 seconds Time Since Wake: 130000 seconds

Crashed Thread: 0 Dispatch queue: com.apple.main-thread

Exception Type: EXC_BAD_ACCESS (SIGSEGV) Exception Codes: KERN_INVALID_ADDRESS at 0x0000000000000110

VM Regions Near 0x110: --> __TEXT 0000000101cb0000-0000000101cb2000 [ 8K] r-x/rwx SM=COW /usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python

Application Specific Information: crashed on child side of fork pre-exec ...

oam31
  • 95
  • 5
  • 1
    What does "doesn't work" mean? What happens? – abarnert May 24 '15 at 20:34
  • As a side note, you don't often want to use `map_async`. If you just want to wait until all the results are done, as in this case, you can just use `map`. If you want to get results as they come in, use `imap` if you care about the order, `imap_unordered` if you don't. The only time `async_map` is really helpful is when you want to kick off multiple map calls and then wait for them all. – abarnert May 24 '15 at 20:37
  • @abarnert A fatal error ! – oam31 May 24 '15 at 20:47
  • @abarnert If I modify asyncResult = pool.map_async(sol2, xrange(1)) to Result = pool.map(sol2, xrange(1)), I have the same problem. – oam31 May 24 '15 at 20:51
  • 1
    What does "a fatal error" mean? Did you get an exception? If so, [edit] your question, and paste in the exception (the whole thing, with traceback). Did you get a segfault or other crash? Then put that in your question. Don't make us keep guessing. – abarnert May 24 '15 at 20:51
  • Meanwhile, I wasn't expecting `map` to solve the problem—that's why I wrote a comment that starts off "As a side note…" It's something you probably want to know about, but it's probably not relevant to solving your current issue. – abarnert May 24 '15 at 20:52
  • From your edit, I don't understand what "crash without exception or error" means. If there's no error, how do you know it crashed? What actually happened? – abarnert May 24 '15 at 20:55
  • @abarnet Try it and see the result ! – oam31 May 24 '15 at 21:07
  • 1
    OK, the problem is that you're trying to implicitly share global arrays between processes. You can't do that. You can manually wrap them up with `np.ctypeslib` and `mp.sharedctypes`, or you can find a wrapper library (IIRC, it's called `numpy-sharedmem`, but isn't in PyPI…) that makes it a little easier. But first make sure that's what you really want. As soon as you have multiple worker processes, you're going to need locking, and most likely between lock-serialization and false sharing you're going to end up with slower code than if you'd just passed the arrays or stored them in files. – abarnert May 24 '15 at 21:32

0 Answers0