1

I have two objects 

shared_array = numpy.zeros(20)

and 

shared_matrix = scipy.sparse.csr_matrix((data, (row, col)),(20,20),dtype=numpy.float64)

which I want to be accessible from all processes created with

multiprocessing.Process(target = random_function, args = (shared_matrix, shared_array))

How is this done?

Chicony
  • 393
  • 3
  • 17

1 Answers1

2

If you just want to access it, you can. You can read data from global variables across all processes created by multiprocessing.

However if you want to write into for example a dictionary (with caution not to overwrite) you can use shared memory objects. multiprocessing has a built in manager from where you can call all the shared memory objects such as lists, dicts.

You pass all your objects in arguments, so it will be available for all of the processes, however if you make changes inside the object it wont we permanent, as it would not be permanent with a simple function.

Simple example for a numpy array:

import numpy as np
import multiprocessing

a = np.zeros(5)

def func(b):
    ns.array += b
    return

manager = multiprocessing.Manager()
ns = manager.Namespace()
ns.array = np.zeros(5)
pool = multiprocessing.Pool(2)

list(pool.imap_unordered(func, [1,2]))
print(ns.array)

Will output [ 3. 3. 3. 3. 3.]

Here is another very detailed solution: https://stackoverflow.com/a/7908612/4555249

Community
  • 1
  • 1
Gábor Erdős
  • 3,599
  • 4
  • 24
  • 56
  • I want to be able to read and write. I've seen the manager but do they work with `numpy.array` and `scipy.sparse.csr_matrix` ? – Chicony Dec 07 '16 at 12:25
  • As i wrote you can read all global variables inside each process spawned by `multiprocessing`. You dont even have to pass them in as arguments. – Gábor Erdős Dec 07 '16 at 12:25
  • I ll add a simple example in my answer – Gábor Erdős Dec 07 '16 at 12:34
  • That example will output `[3,3,3,3,3]` *sometimes*. Because the array is not locked it can happen that there is a race condition and only one of the two function applications gets commited. The result can as well be `[1,1,1,1,1]` or `[2,2,2,2,2]`, and maybe even anything in between (although I haven't seen it) – Zah Dec 25 '16 at 10:48
  • Manager should take care of the lock. I belive you will never see any other output then the 3333 – Gábor Erdős Dec 26 '16 at 08:00