0

How do you hstack/vstack a multiprocessing.Array object that is numpy array without overwriting mp.Array object. 

import multiprocessing as mp
import numpy as np 

    def init(aa, vv):
        global a, v
        a = aa
        v = vv

    def worker(i):
        a[i] = v.value * i
        # this next line writes over the mp.Array object. 
        a    = np.vstack([a[:], np.array(range(N) , dtype=np.int64)])

if __name__ == "__main__":
        N = 5
        a = mp.Array('d', np.array(range(N) , dtype=np.int64))
        v = mp.Value('i', 3)
        p = mp.Pool(initializer=init, initargs=(a, v))
        p.map(worker, range(N))
        print(a[:])
Merlin
  • 24,552
  • 41
  • 131
  • 206
  • The `mp.Array` has nothing to do with `numpy` array, though it can be initialed with values from `np.array`. From a quick scan of the multiprocessing docs, I'd say `a` is initialed to a memory block of `N` floats. It's size cannot be changed by the worker. You can write new values to it, but you can't add to it. – hpaulj Sep 06 '16 at 00:30
  • What are you actually trying to do? Considering taking a step back and asking a question about the higher lever problem you're trying to solve. Using mp.Array with numpy and vstack is probably not the right approach to that problem. – Bi Rico Sep 06 '16 at 00:49
  • @BiRico What I trying to do is: I have a calc intensive protocol which I multi-process then insert into database at the child level. What I am trying to do is create a table of sorts at the parent level. Do the heavy calc protocol, (get lock) update the table, then after the child processes finish. update the database. My attempt at mp.Array was to see if it could serve as the temp table. – Merlin Sep 06 '16 at 01:23
  • It does not look like `mp.Array` has any notion of multidimensionality. – hpaulj Sep 06 '16 at 02:18
  • The type of thing you described is usually done by having the worker thread return a row/entry of the final table. Then you can simple do `table = mp.map(f, data)`. Is there a reason you're trying to have the worker update the table instead of simply returning the result? – Bi Rico Sep 06 '16 at 04:11
  • http://stackoverflow.com/questions/7894791/use-numpy-array-in-shared-memory-for-multiprocessing may help – hpaulj Sep 06 '16 at 04:22

0 Answers0