3

I am wondering if I create a dict through multiprocessing.Mananger(), will its value be locked automatically when a processing is manipulating it, or I should write lock.acquire/release explicitly?

Previously, I wrote lock.acquire/release inside the function explicitly, however, it seems my code suffer from the problem of dead lock. It is strange since I think there is only one lock in my code. Therefore I am wondering if manager.dict will give another lock automatically. When I delete the lock.acquire/release, the code works fine. But I am not sure if the dict is correct.

import multiprocessing as mp
from functools import partial

def function(file_name, d, lock):
    key, value = read_files(file_name)
    #lock.acquire()
    if (key not in d):
        d[key] = []
    tmp = d[key]
    tmp.append(value)
    d[key] = tmp
    #lock.release()

if __name__ == '__main__':
    manager = mp.Manager()
    d = manager.dict()
    lock = manager.Lock()
    partial_function = partial(function, d=d, lock=lock)
    pool = mp.Pool(10)
    pool.map(partial_function, files) #files is a predefined list
    pool.close()
    pool.join()

Some of the related questions are listed below, but they seems to contradict to each other.

Why a manager.dict need to be lock before writing inside?

How python manager.dict() locking works:

Peter Go
  • 31
  • 1
  • With dummy definitions of `read_files` and `files`, [I get no deadlock](https://ideone.com/IoVnl1). You have some bug that didn't make its way into your question. – user2357112 Mar 03 '22 at 00:26

1 Answers1

0

Yes, "locking'' occurs automatically with multiprocessing.Manager().dict(). It is thread-safe in the sense that the internals will only allow one process to access (read or write) the object at any given time.

The Skylur
  • 13
  • 1
  • 2
  • This is technically true, but the locking is at the level of individual dict operations, not at the level the code in the question needs. The code in the question needs to do its own locking. – user2357112 Mar 03 '22 at 00:28