3

I would like to be able to send shared arrays to processes. I'm looking for a way which enables me to re-use processes when the shape of the shared arrays keeps changing. The code should be able to run interactively, hence I prefer to avoid Pool. Unfortunately it seems I cannot send RawArray objects using multiprocessing.Pipe. Is there some way to avoid this problem?

import multiprocessing as mp
from pylab import *

N = 100

Z = mp.RawArray('d', randn(100))

def process_tasks(conn):
    while True:
        task = conn.recv()
        if task is False:
            conn.send(True)
        else:
            new_Z = task[0]
            print sum(array(new_Z))
            conn.send(True)

home_pipe, away_pipe = mp.Pipe()
my_proc = mp.Process(target=process_tasks, args=(away_pipe,))
my_proc.start()

home_pipe.send((Z,))

if home_pipe.recv():
    print "Done!"

This results in:

PicklingError                             Traceback (most recent call last)
<ipython-input-4-0c6a5494376a> in <module>()
     20 my_proc.start()
     21 
---> 22 home_pipe.send((Z,))
     23 
     24 if home_pipe.recv():

PicklingError: Can't pickle <class 'multiprocessing.sharedctypes.c_double_Array_100'>: attribute lookup multiprocessing.sharedctypes.c_double_Array_100 failed
conjectures
  • 801
  • 2
  • 7
  • 24
  • you cannot send `RawArray` via pipe, you have to inherit `RawArray` in newly created `mp.Process` (`maxtasksperchild`), [code example](http://stackoverflow.com/q/7894791/4279). – jfs Sep 14 '15 at 16:31
  • 1
    I understand how to do that, but do not wish to do it for the reasons outlined above. I'm looking for a a way which enables process re-use without Pool. – conjectures Sep 14 '15 at 16:47
  • I understand that is why I've posted it as a comment (for other readers), not an answer for you. – jfs Sep 14 '15 at 16:49
  • @conjectures did you ever figure this out? I have same question – user-2147482637 May 08 '19 at 20:14
  • No, at the time I just switched to Julia for this and other reasons. I understand that's not always an option. – conjectures May 09 '19 at 10:51

0 Answers0