2

I would like to store a 2-d array as shared memory array, so each process created by multiprocessing module can access the 2-d array. But it seems only vector is allowed. If I change the shape in the following example from (6,1) to (2,3), the example won't work and it says "TypeError: only length-1 arrays can be converted to Python scalars"

from multiprocessing import Process, Value, Array
import numpy as np

def f(n,a):
    a[:] = np.ones(shape=(6,1), dtype=float)[:]

if __name__ == '__main__':
    tmp =  np.zeros(shape=(6,1), dtype=float)
    print tmp

    arr = Array('f', tmp)

    p1 = Process(target=f, args=(1,arr))
    p1.start()
    p1.join()

    print 'done'
    print arr[:]
read Read
  • 5,765
  • 4
  • 29
  • 30

1 Answers1

3

This is because mutiprocessing.Array is a wrapper around python's array type, not numpy.ndarray and python's array type doesn't support multiple dimensionality. Specifically, look at the documentation for the initializer:

A new array whose items are restricted by typecode, and initialized from the optional initializer value, which must be a list, string, or iterable over elements of the appropriate type.

You've got an iterable (a multi-dimensional array), but it yields views/arrays, not elements of the appropriate type.

mgilson
  • 300,191
  • 65
  • 633
  • 696
  • Is there a quick way to convert numpy.ndarray to array? – read Read Apr 30 '14 at 14:55
  • @readRead -- Only 1D numpy arrays can be converted to `array` -- and it makes a copy (not a view) – mgilson Apr 30 '14 at 14:57
  • Thanks. I plan to do some matrix manipulation on the shared array in each process. If I just store 1-d array, it's not very convenient. How can I shared a 2-d numpy array across processes and avoid copy? maybe use something other than multiprocessing.Array? – read Read Apr 30 '14 at 15:03
  • I'm not entirely convinced it's possible. As far as I see it, the best you can do is create a wrapper around the `Array`/`array` which provides a nicer interface. – mgilson Apr 30 '14 at 15:06
  • If I create a wrapper around Array/array, which converts 1-d array to 2-d numpy.ndarray, does this involves extra copy? – read Read Apr 30 '14 at 15:09
  • @readRead -- Yep. That would involve a copy. As far as I know, there's no way for a numpy.ndarray and an array.array to share data. – mgilson Apr 30 '14 at 15:16