Numpy parallelization for more than one node?

Question

It is possible to use more cores for certain numpy operations like np.dot. Is it also possible to use more than one node?

The answer on this page is "If you need sophisticated parallelism - you have a computing cluster, say, and your jobs need to communicate with each other frequently - you will need to start thinking about real parallel programming. " Apparently there is not answer on this page? — varantir, Jul 08 '15 at 13:45
Your build may already be using multiple cores. See http://stackoverflow.com/q/5991014/553404 and http://stackoverflow.com/q/5260068/553404 — YXD, Jul 08 '15 at 13:49

score 3 · Answer 1 · edited May 23 '17 at 10:26

3

Numpy is not designed for easy splitting into multiple nodes. You have to perform the split manually, i.e. split into subarrays processed by different nodes (if possible at all for your operation).

You may be using multiple cores, depending on the underlying library [1].

Alternatively, you can look at Blaze and Dask modules for Numpy type of operations with multicore support.

edited May 23 '17 at 10:26

Community

1
1

answered Jul 08 '15 at 13:48

Eriks Dobelis

913
7
16

This blogpost describes using dask to parallelize matmul operations. http://matthewrocklin.com/blog/work/2015/01/14/Towards-OOC-MatMul/ But really you should look into distributed BLAS implementations. The ScaLAPACK or Elemental libraries might be of use. – MRocklin Jul 12 '15 at 08:44

score 0 · Answer 2 · edited May 23 '17 at 12:14

For splitting work on several process you have to implement Process (not Thread)!

Example (extract from official doc):

from multiprocessing import Process

def f(name):
    print('hello', name)

if __name__ == '__main__':
    p = Process(target=f, args=('bob',))
    p.start()
    p.join()

See for detail the official documentation

Additionally, this topic may be of interest regarding the numpy optimized syntax related to your situation.

EDIT As stated on this link, you can use this to split over cores or nodes!

Numpy parallelization for more than one node?

2 Answers2