How to sync a variable with multiple processes in python?

Question

I wouldd like to use multiprocessing pool to access, read, and modify a variable in python. The processes are completely separate, but have to access the same variable. This would be easily done with pointers in C, but how do I do it in python?

For example, I have done this:

import multiprocessing as mp
import time
a=3

def one(arg):
        time.sleep(2*arg[1])
        print(arg)
        arg[0]+=1
        print(arg[0])
        return arg[0]

if __name__ == '__main__':
        with mp.Pool(processes=2) as pool:
                print(pool.map(one,[[a,1],[a,2]]))

But it returns

[3, 1]

4

[3, 2]

4

[4, 4]

Instead of [4,5].

The `multiprocessing` package spawns separate processes. It has an API similar to the `threading` module, but unlike threads, processes have their own address space. — Adrian W, May 01 '19 at 11:02
It depends on what you want to do. If you really want separate processes, you need to use the shared memory features as described in [this answer](https://stackoverflow.com/a/55936263). If you use plain [threads](https://docs.python.org/3/library/threading.html), then you only need to synchronize access with a [`Semaphore`](https://docs.python.org/3/library/threading.html#semaphore-objects). There is also a thread pool in the `multiprocessing` module. See [this question](https://stackoverflow.com/q/3033952) — Adrian W, May 01 '19 at 14:05

CodeSamurai777 · Accepted Answer · 2019-05-01T13:26:24.837

You can share variables between processes. Using shared memory.

from multiprocessing import Process, Value, Array

def f(n, a):
    n.value = 3.1415927
    for i in range(len(a)):
        a[i] = -a[i]

if __name__ == '__main__':
    num = Value('d', 0.0)
    arr = Array('i', range(10))

    p = Process(target=f, args=(num, arr))
    p.start()
    p.join()

    print(num.value)
    print(arr[:])

Value and Array are special objects that allows you to share state between them. The d and i are typcodes of variable type you want to share.

However, this solution doesn't provide any sync mechanism. You need to be careful not to read and write at the same time.

Also this solution can be used only for simple values. If you need to share object you should look at Manager class in multiprocessing module.

An exert from docs

A manager returned by Manager() will support types list, dict, Namespace, Lock, RLock, Semaphore, BoundedSemaphore, Condition, Event, Barrier, Queue, Value and Array

So as you see you can also share RLock to set up a sync mechanism for reading and writing your shared variables.

More info here: https://docs.python.org/3.5/library/multiprocessing.html#sharing-state-between-processes

Yeah, but I'm using it in selectors in sockets that modify one single variable in the RAM, so I don't know whether it would work here. — Aphrodite, May 02 '19 at 01:02

How to sync a variable with multiple processes in python?

1 Answers1