Value is not storing into Manager().dict() inside the Pool()

Question

This is the main code:

if __name__ == '__main__':
    manager = Manager()
    mylist = manager.list()
    mydict = manager.dict()

    mylist.append('abc')
    mylist.append('xyz')

    for k in mylist:
        mydict[k] = manager.list()

    pool = Pool(processes = 100)
    pool.map(func, arg)
    pool.terminate()

This is the func() definition:

for x in mylist:
    r = {}
    r['m'] = 1
    r['n'] = 2
    print (r) # gives correct values
    mydict[x].append(dict(r))
    print (mydict) # gives empty list, keys are printed correct

As commented in the code, it is printing empty list instead of expected values. How do I make this shared dict and list work properly?

First print is showing expected values. Second print shows expected keys but the value part is empty list. Since it shows the keys correctly, the sharing is working. There is some issue with the append part. I'm unable to figure it out.

Down voters, please do read the linked question carefully before you assume that my question is same as that one. If you still insist they are the same, please leave a comment stating why you think so. From my viewpoint, both questions are different. Linked question asks about how to share a dict with answers suggesting use of Manager.dict(). My question already uses Manager.dict() and is failing because of an anomaly. Thank you.

[Edit] your Question and show the function header of `func(???`, or do you use `mylist` as a Global? — stovfl, Sep 05 '17 at 08:47
@pdshah There is supposed to be only one instance of mydict and one instance of mylist in whole program. Both are global. All data inside mydict must be visible and modifiable from myfunc(). I do not understand what you mean by "all manager instances". Am I creating more instances inside the pool? That is not intended. Help me fix it. — aste123, Sep 05 '17 at 09:26
@stovfl yes mylist is global. From what I understand if I declare manager.list() as global then it will be shared with the pool. I have declared it in the main part of the script. So, I hope it is global. Help me fix it if it is wrong. — aste123, Sep 05 '17 at 09:27
I have given the definition of func() also. I do not understand what you mean by showing the function header. — aste123, Sep 05 '17 at 09:29
Possible duplicate of [Python multiprocessing: How do I share a dict among multiple processes?](https://stackoverflow.com/questions/6832554/python-multiprocessing-how-do-i-share-a-dict-among-multiple-processes) — stovfl, Sep 05 '17 at 10:04
@stovfl No, the linked question doesn't answer my question. The linked question suggests using Manager.dict. I'm already using it in my code. My question is not about how to share a normal dict. My question is asking why sharing dict using manager is not working. — aste123, Sep 06 '17 at 01:19
Down voters, please do read the linked question carefully before you assume that my question is same as that one. Thank you. — aste123, Sep 06 '17 at 01:19

aste123 · Accepted Answer · 2017-09-06T01:12:35.693

2

As can be seen from the comments in the code, mydict is correctly read inside func() but fails to mutate it. Manger.dict() object can't mutate nested parts.

It can be fixed by creating a new dictionary and pointing mydict to it after mutation.

for x in mylist:
    r = {}
    r['m'] = 1
    r['n'] = 2
    print (r)
    foo = mydict[x]
    foo.append(dict(r))
    mydict[x] = foo
    print (mydict)

edited Sep 06 '17 at 01:12

answered Sep 05 '17 at 10:00

aste123

1,223
4
20
40

Value is not storing into Manager().dict() inside the Pool()

1 Answers1