Shared array of objects using multiprocessing in python

Question

I am trying to create an array of shared objects (instances of a class) between different processes in a python program. Each of these objects would be modified in the program. For this purpose I am using multiprocessing as follows:

import multiprocessing
import numpy as np
sh = multiprocessing.RawArray (ctypes.py_object, 10)
f = np.frombuffer(sh, dtype=object)

and the error I get is:

Traceback (most recent call last):
  File "<pyshell#14>", line 1, in <module>
    f = np.frombuffer(sh, dtype=object)
ValueError: cannot create an OBJECT array from memory buffer

Now my question is, first of all, is this generally the right way to tackle this problem, and second, what is my mistake in the above code? Thank you in advance.

score 3 · Accepted Answer · edited May 23 '17 at 10:33

3

ctypes.py_object represents a PyObject * in C. It is a pointer to a struct that represents a Python object, resides in the private memory of the process, and contains more pointers. Another process can't access it; trying to share a pointer between processes is not useful.

See also this answer by Alex Martelli.

edited May 23 '17 at 10:33

Community

1
1

answered Nov 07 '14 at 13:56

Janne Karila

24,266
6
53
94

Actually, using Array and RawArray (in multiprocessing) is a method for creating a shared array in the memory to be accessed by multiple processes. I can write the above code for lets say a shared array of integers but have problem writing it for an array of objects. For example if I write: >>> import multiprocessing as mp >>> import numpy as np >>> sh = mp.RawArray (ctypes.c_int , 10) >>> f = np.frombuffer(sh, dtype=int) >>> f[0]=1 >>> sh[0] 1 It works flawlessly! – user823743 Nov 07 '14 at 15:13
@user823743 Correct, but you can't do the same with pointers. – Janne Karila Nov 07 '14 at 18:04
You are correct; fortunately I found another way to avoid multiprocessing all together. Thanks. – user823743 Nov 08 '14 at 19:24

score 0 · Answer 2 · answered May 29 '17 at 09:54

You might want to use a multiprocessing.Queue, in which you can just dump objects without worrying about types. It's also thread-safe and process-safe.

Here's a simple example of a Queue used to facilitate the Producer-Consumer problem (original source, Pizzas were a little bonus by me).

from multiprocessing import Process, Queue

class Pizza(object):
    def __init__(self, pizza_num):
        self.pizza_num = pizza_num
        self.num_slices = 8

sentinel = "NO PIZZA"

def producer(initial_num_pizzas, total_num_pizzas, q):
    """Cooks Pizzas to be consumed and waits for the consumer to finish eating."""
    print("Producer: I am cooking %s Pizzas and putting them on the Queue!"%(total_num_pizzas-initial_num_pizzas))
    for i in range(q.qsize(), total_num_pizzas):
        print("Producer: Behold, for I have cooked Pizza no. %s"%i)
        q.put(Pizza(i))
    q.put(sentinel)

def consumer(q):
    """Consumes some Pizza. In this case, all it does is set the number of slices to 0."""
    while True:
        pizza = q.get()
        pizza.num_slices = 0
        if pizza == sentinel:
            break
        print("Comsumer: Pizza no. %s was found! It has %s slices, yum!"%(pizza.pizza_num, pizza.num_slices))

if __name__ == '__main__':
    q = Queue()
    total_num_pizzas = 10
    initial_num_pizzas = 4
    ## Let's add some Pizzas beforehand:
    for i in range(0, initial_num_pizzas):
        q.put(Pizza(i))
    print("Main: I have precooked %s Pizzas."%q.qsize())

    producer_proc = Process(target=producer, args=(initial_num_pizzas, total_num_pizzas, q))
    consumer_proc = Process(target=consumer, args=(q,))
    producer_proc.start()
    consumer_proc.start()

    q.close()  ## Shop is closed, no more Pizzas will be added to Queue!
    q.join_thread()

    producer_proc.join()
    consumer_proc.join()

Below is an example output. If you run it, the Producer and Consumer print statements may be interleaved differently, because of non-deterministic execution of parallel processes.

Main: I have precooked 4 Pizzas.
Producer: I am cooking 6 Pizzas and putting them on the Queue!
Producer: Behold, for I have cooked Pizza no. 4
Producer: Behold, for I have cooked Pizza no. 5
Producer: Behold, for I have cooked Pizza no. 6
Producer: Behold, for I have cooked Pizza no. 7
Comsumer: Pizza no. 0 was found! It has 8 slices, yum!
Comsumer: Pizza no. 1 was found! It has 8 slices, yum!
Producer: Behold, for I have cooked Pizza no. 8
Comsumer: Pizza no. 2 was found! It has 8 slices, yum!
Producer: Behold, for I have cooked Pizza no. 9
Comsumer: Pizza no. 3 was found! It has 8 slices, yum!
Comsumer: Pizza no. 4 was found! It has 8 slices, yum!
Comsumer: Pizza no. 5 was found! It has 8 slices, yum!
Comsumer: Pizza no. 6 was found! It has 8 slices, yum!
Comsumer: Pizza no. 7 was found! It has 8 slices, yum!
Comsumer: Pizza no. 8 was found! It has 8 slices, yum!
Comsumer: Pizza no. 9 was found! It has 8 slices, yum!

Note that you should use Sentinels to mark the end of your Queue. I've used "NO PIZZA" here, but they can be anything at all.

Shared array of objects using multiprocessing in python

2 Answers2