Unable to update nested dictionary value in multiprocessing's manager.dict()

Question

I am trying to update a key in a nested dictionary of multiprocessing module's manager.dict() but not able to do so. It doesn't update the value and doesn't throw any error too.

Code:

import time
import random
from multiprocessing import Pool, Manager

def spammer_task(d, token, repeat):
    success = 0
    fail = 0
    while success+fail<repeat:
        time.sleep(random.random()*2.0)
        if (random.random()*100)>98.0:
            fail+=1
        else:
            success+=1
        d[token] = {
            'status': 'ongoing',
            'fail': fail,
            'success': success,
            'repeat': repeat
        }
    print d[token]['status']
    d[token]['status'] = 'complete'
    return

p = Pool()
m = Manager()
d = m.dict()

p.apply_async(spammer_task (d, 'abc', 5))
print d

Output:

ongoing

{'abc': {'status': 'ongoing', 'fail': 0, 'repeat': 5, 'success': 5}}

My expectations are that as soon as while loop ends, it should make d['abc']['status'] = complete. But on final print it prints its status as 'ongoing' only.

See also: [Creating and updating nested dictionaries and lists inside multiprocessing.Manager object](https://stackoverflow.com/q/73409151/16310741) — Charchit Agarwal, Aug 20 '22 at 21:10

domoarigato · Accepted Answer · 2020-06-19T11:35:54.067

15

not sure why, but the Manager DictProxy object can't seem to handle mutating a nested part. this code works:

import time
import random
from multiprocessing import Pool, Manager

def spammer_task(d, token, repeat):
    success = 0
    fail = 0
    while success+fail<repeat:
        time.sleep(random.random()*2.0)
        if (random.random()*100)>98.0:
            fail+=1
        else:
            success+=1
        d[token] = {
            'status': 'ongoing',
            'fail': fail,
            'success': success,
            'repeat': repeat,
        }
    print d[token]['status']
    foo = d[token]
    foo['status'] = 'complete'
    d[token] = foo
    return

p = Pool()
m = Manager()
d = m.dict()

p.apply_async(spammer_task(d, 'abc', 5))
print d

edited Jun 19 '20 at 11:35

answered May 29 '16 at 13:24

domoarigato

2,802
4
24
41

Thanks, no other solution that using additional memory? I am going to deamonize this multiprocessor inside a HTTP Server and dont feel like handling additional memory + cleanup for them. – MohitC May 29 '16 at 13:44
According to the note under this part of the docs: https://docs.python.org/2/library/multiprocessing.html#using-a-remote-manager - the above seems to be the suggested way to doing this. – domoarigato May 29 '16 at 13:53
The note that @domoarrigato posted is out-of-date - but it's true that the official Python documentation states this answer to be correct. Here is the new link - scroll down just a tad to find the 'Note': https://docs.python.org/2/library/multiprocessing.html#managers – Dan Nissenbaum Feb 25 '18 at 03:16
As of today, it's still the case in python3. Just scroll down a little and check the subsection before the [note](https://docs.python.org/3/library/multiprocessing.html#proxy-objects) with the search phrase "`contained in a referent`". – Brainor Aug 20 '22 at 16:58
This is the fastest fix to the problem, but may cause issues if the dictionary needs to be synchronized. For example, process A retrieves dict from manager and makes changes to it in local memory; now for this small duration the managed dictionary and the one that exists in A are different, and if another process attempts to access it, it will get an outdated copy of the dict. If this is a nuisance to you, then you should use locks, or consider [this alternate solution](https://stackoverflow.com/a/73418403/16310741) which automatically creates managed objects for nested dictionaries/lists – Charchit Agarwal Aug 20 '22 at 21:04
2 years later no fix – greendino Jul 03 '23 at 09:06

score 5 · Answer 2 · edited Feb 06 '18 at 16:37

Looks like this issue remains per code below:

import multiprocessing, sys;

if __name__ == '__main__':

print(sys.version);

mpd = multiprocessing.Manager().dict();
mpd['prcss'] = {'q' : 'queue_1', 'ctlg' : 'ctlg_1' };

# update 1 - doesn't work!
mpd['prcss'].update( { 'name': 'concfun_1'} );
print('Result of failed update 1:', mpd['prcss']);

# update 2 - doesn't work!
mpd['prcss']['name'] = 'concfun_1';
print('Result of failed update 2:', mpd['prcss']);

# update 3 - works!
mpd_prcss = mpd['prcss'];
mpd_prcss['name'] = 'concfun_1';
mpd['prcss'] = mpd_prcss;
print('Result of successful update 3:', mpd['prcss']);

Output:

3.6.1 (v3.6.1:69c0db5, Mar 21 2017, 17:54:52) [MSC v.1900 32 bit (Intel)]

Result of failed update 1: {'q': 'queue_1', 'ctlg': 'ctlg_1'}

Result of failed update 2: {'q': 'queue_1', 'ctlg': 'ctlg_1'}

Result of successful update 3: {'q': 'queue_1', 'ctlg': 'ctlg_1', 'name': 'concfun_1'}

Unable to update nested dictionary value in multiprocessing's manager.dict()

2 Answers2

Linked