0

the matrix Z is defined like this:

N = 15
Z = np.zeros((N, N))

and I want different threads to fill it so I'm need to create a shared memory matrix or array. I'm doing this:

Z_shared = Array('d', Z)

But I get this error:

TypeError: only length-1 arrays can be converted to Python scalars

Which I suppose means that Array doesn't accept matrices. Is there another function that does what I need or should I convert Z to a vector first and then pass it to Array. If so is there a function that does this for me. I don't want to loop over Z since N can be very large.

I'm fill Z_shared in another function that is called like this:

from multiprocessing import Pool, Array
pool = Pool(processes=1)

def foo(data):
    x, y, val = data
    Z_shared[x][y] = val
pool.map(fooe, DATA)

'data' is a tuple of indexes and values and 'DATA' is a list a of tuples.

thank you

soroosh.strife
  • 1,181
  • 4
  • 19
  • 45
  • The second parameter to Array is a size. Edit: wait. no. – leewz Feb 03 '14 at 08:21
  • Check this question out: http://stackoverflow.com/questions/17785275/share-large-read-only-numpy-array-between-multiprocessing-processes – leewz Feb 03 '14 at 08:27
  • The processes that generate the data don't touch the same entries, right? Maybe use the recommended http://docs.python.org/2/library/multiprocessing.html#pipes-and-queues somehow. – leewz Feb 03 '14 at 08:42
  • No each process fills only one element of Z – soroosh.strife Feb 03 '14 at 08:43
  • Actually reading your foo... If it's that simple, why are you even using multiprocessing? Creating processes is expensive. Or is foo more conplex? – leewz Feb 03 '14 at 08:46
  • foo is not really complex but it is being called more than a couple of thousand times – soroosh.strife Feb 03 '14 at 10:35
  • also i'm dealing with a lot of memory access here – soroosh.strife Feb 03 '14 at 11:16
  • That won't really be helped by lots of multiprocessing. If the x and y pairs are unique, run a fixed number of processes that each fill in their own matrices, then add them together with a numpy array summation (each cell will be nonzero in at most one result). – leewz Feb 03 '14 at 16:24

1 Answers1

1

Strictly just answering the question, if the object constructed by Array acts like a list, you can copy everything like so:

Z_shared = Array('d', Z.size)
Z_shared[:] = Z.reshape((-1)) # copy entrywise

That seems bad, though. For one thing, the shared array won't have ndarray semantics, and will be a copy. Does it really need to be a single shared variable?

You're splitting into N*N processes in your program, where N can be large. Making a new process can be expensive. You should only split into as many processes as you have cores, I think.

So the idea is, split the data based on the x coordinate, and have one process to deal with the top half of the matrix and the other to deal with the bottom half. Both would be separate arrays in shared memory.

Reading into your question, you should not be playing with multiprocessing in the first place (premature optimization!). Instead, check out sparse matrices: http://docs.scipy.org/doc/scipy/reference/sparse.html

leewz
  • 3,201
  • 1
  • 18
  • 38