0

This question following this one [1]. I have a big 3D array and i have to do some heavy calculations on it. I would like to split a slice of my array in 4 parts and do calculations for each part with each 4 cores of my computer... And do that for each slices of my 3D array...what is the best way to do that? 

import numpy

size = 8.
Y=(arange(2000))
X=(arange(2000))
(xx,yy)=meshgrid(X,Y)

array=zeros((Y.shape[0],X.shape[0],size))

array[:,:,0] = 0
array[:,:,1] = X+Y
array[:,:,2] = X*cos(X)+Y*sin(Y)
array[:,:,3] = X**3+sin(X)+X**2+Y**2+sin(Y)
Community
  • 1
  • 1
user3601754
  • 3,792
  • 11
  • 43
  • 77

2 Answers2

2

You can use Pool from the multiprocessing module:

from multiprocessing import Pool

def f(num):
  return num * 2 # replace with heavy computation

lst = [1,2,3,4,5,6,7,8,9,10,11]
p = Pool(4)
print p.map(f, lst)

It will work equally well with a 3-dimensional numpy array:

from multiprocessing import Pool
import numpy

def f(num):
  return num * 2 # replace with heavy computation

arr = numpy.array(
  [numpy.array([
    numpy.array([1,2,3]),
    numpy.array([4,5,6]),
    numpy.array([7,8,9])]),
   numpy.array([
     numpy.array([1,2,3]),
     numpy.array([4,5,6]),
     numpy.array([7,8,9])])])
p = Pool(4)
print p.map(f, arr)
EvenLisle
  • 4,672
  • 3
  • 24
  • 47
  • Thanks for your help, but when i calculate the time with and without multiprocess...i find that without multiprocess i m faster :s – user3601754 Apr 22 '15 at 11:29
  • @user3601754 How expensive is each call to `f`? If `f` doesn't take a significant amount of time to run, then `multiprocessing` isn't going to help, because the overhead of passing the array contents between processes will be greater than the benefit of running 4 instances of `f` simultaneously. – dano Apr 22 '15 at 14:27
1

As an alternative to multiprocessing, you can use the concurrent.futures module:

import concurrent.futures

def f(num):
    return num * 2
arr = […]

with concurrent.futures.ProcessPoolExecutor() as exc:
    print(list(exc.map(f, arr)))
Arthur
  • 4,093
  • 3
  • 17
  • 28
  • Thanks for your help! I try that but when i print i get "" and i dont find how to read my results :s – user3601754 Apr 28 '15 at 13:27
  • See the doc on python generators: https://wiki.python.org/moin/Generators. You can iterate over the results of `exc.map()` with a for loop – Arthur Apr 30 '15 at 10:37
  • By far the simplest way to do "do something on every item of this array, do it in parallel". Worth mentioning that [ProcessPoolExecutor() will default to the number of processors in the machine](https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ProcessPoolExecutor) – H.Scheidl Sep 18 '17 at 21:05