Can I create a shared multiarray or lists of lists object in python for multiprocessing?

Question

I need to make a shared object of a multidimensional array or list of lists for it to be available to the other processes. Is there a way to create it as for what i have seen it is not possible. I have tried:

from multiprocessing import Process, Value, Array
arr = Array('i', range(10))
arr[:]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
arr[2]=[12,43]
TypeError: an integer is required

I heard numpy array can be multiarray and a shared object, if above is not possible can someone tell me how to make a numpy array a shared object??

I'm not sure about the multiprocessing part, but I know why you're receiving a TypeError there: You initialized your array for integer values in the 2nd line. In the 5th line, you try to assign a list to a point in the array. Arrays are relatively low level objects and can't hold lists, dicts, sets and other python container types. — Joel Cornett, Mar 17 '12 at 21:56
Yes that is what i know, i know why i am getting the TypeError. That is why i am asking if is there a way to overpass this issue, or create a shared object "List", or maybe how to use numpy Arrays that are able to be multiarrays and make them shared objects. — user1249212, Mar 17 '12 at 22:07
Review these. Then, build a data structure around them: http://effbot.org/pyfaq/what-kinds-of-global-value-mutation-are-thread-safe.htm. — MrGomez, Mar 17 '12 at 23:16

score 33 · Accepted Answer · edited May 23 '17 at 11:46

To make a numpy array a shared object (full example):

import ctypes as c
import numpy as np
import multiprocessing as mp

n, m = 2, 3
mp_arr = mp.Array(c.c_double, n*m) # shared, can be used from multiple processes
# then in each new process create a new numpy array using:
arr = np.frombuffer(mp_arr.get_obj()) # mp_arr and arr share the same memory
# make it two-dimensional
b = arr.reshape((n,m)) # b and arr share the same memory

If you don't need a shared (as in "share the same memory") object and a mere object that can be used from multiple processes is enough then you could use multiprocessing.Manager:

from multiprocessing import Process, Manager

def f(L):
    row = L[0] # take the 1st row
    row.append(10) # change it
    L[0] = row #NOTE: important: copy the row back (otherwise parent
               #process won't see the changes)

if __name__ == '__main__':
    manager = Manager()

    lst = manager.list()
    lst.append([1])
    lst.append([2, 3])
    print(lst) # before: [[1], [2, 3]]

    p = Process(target=f, args=(lst,))
    p.start()
    p.join()

    print(lst) # after: [[1, 10], [2, 3]]

From the docs:

Server process managers are more flexible than using shared memory objects because they can be made to support arbitrary object types. Also, a single manager can be shared by processes on different computers over a network. They are, however, slower than using shared memory.

This seems to be a great solution, i think i will use managers as they seem to be a solution that makes little modification to my preexisting code and the fact that it can even work over a network makes it ideal for the next fase of my project. — user1249212, Mar 18 '12 at 00:15
just one quick question, will managers create a new copy of the list and then synchronize it between processes or will it share the memory from the list? — user1249212, Mar 18 '12 at 00:47
@user1249212: Model it as if processes are always distributed over a network. — jfs, Mar 18 '12 at 03:26

score 2 · Answer 2 · answered Mar 17 '12 at 22:18

2

Why not create a list of Arrays?

 arrays = [Array('i', range(10))] * 10

answered Mar 17 '12 at 22:18

Lev Levitsky

63,701
20
147
175

Actually mooooeeeep mentioned it in the [previous answer](http://stackoverflow.com/a/9743806/1258041) – Lev Levitsky Mar 17 '12 at 22:36
That's because i think there exists no shared object for lists and the way i previously did my code the function must be aware of (at least one of the rows of matrix1 and all the rows of matrix2). I know there are many other ways in which i can easily reorganize my matrix for me not to need sharing multi arrays or lists of lists, but i would like to know all the possibilities on how to do it. – user1249212 Mar 17 '12 at 23:12

Can I create a shared multiarray or lists of lists object in python for multiprocessing?

2 Answers2

Linked