Multiprocessing and dict empty after update [python3]

Question

i have one problem with my code in python3 : i tried many solutions but still the same result.

import pprint
import concurrent
import multiprocessing
from concurrent import futures

class exempleStackOverFlow: 
    def __init__(self):
        self.dict1={}
        self.headers = {"header1","header2","header3"}

    def fillIn(self,key):
        response = {key:{'first':['1','2','3']}}
        self.dict1.update(response)
        pprint.pprint(response)

e =  exempleStackOverFlow()

def try_my_operation(item):
    try:
        e.fillIn(item)      
    except:
        print('error with item')

executor = concurrent.futures.ProcessPoolExecutor(2)
futures = [executor.submit(try_my_operation, item) for item in e.headers]
concurrent.futures.wait(futures)
print(e.dict1)

Result

{'header3': {'first': ['1', '2', '3']}}
{'header1': {'first': ['1', '2', '3']}}
{'header2': {'first': ['1', '2', '3']}}
{}

Expected

 {'header3': {'first': ['1', '2', '3']}}
 {'header1': {'first': ['1', '2', '3']}}
 {'header2': {'first': ['1', '2', '3']}}
 {'header1': {'first': ['1', '2', '3']},
'header2': {'first': ['1', '2', '3']},
     'header3': {'first': ['1', '2', '3']}}

(or what ever order of headers)

I might not have understood how this is working, i would like some light on my problem please, the dict1 is empty at the end when i enter try_my_operation.

it does update well without the multiprocessing/concurrent ( like if i do e.fillIn("headers1"),e.fillIn("headers2")... and after i can get the dict1 full.

Thank you for any future comments

You are creating 2 processes on top of the one main process. Your result is expected from your code. Is the idea to get your current result plus print all three appended together from `self.dict1.update`? — Treyten Carey, Sep 29 '17 at 14:23
Thank for your answer, it s just the result from both print to track changes. in the Result my dict returned is empty {} instead of what i would want in the 'expected' which mean `self.dict1 equal to {'header1': {'first': ['1', '2', '3']}, 'header2': {'first': ['1', '2', '3']}, 'header3': {'first': ['1', '2', '3']}}` instead of self.dict1 equal to {} — M1n1M1n1ng, Sep 29 '17 at 14:28

Treyten Carey · Answer 1 · 2017-09-29T14:43:17.407

I believe you're looking for ThreadPoolExecutor, not ProcessPoolExecutor.

Here I use multithreading (without concurrent) to get your expected output:

import pprint
from threading import Thread

class exempleStackOverFlow: 
    def __init__(self):
        self.dict1={}
        self.headers = ["header1","header2","header3"]

    def fillIn(self,key):
        response = {key:{'first':['1','2','3']}}
        self.dict1.update(response)
        pprint.pprint(response)

def try_my_operation(item):
    try:
        e.fillIn(item)
    except:
        print('error with item')

e =  exempleStackOverFlow()

for i in range(len(e.headers)):
    futures = Thread(target = try_my_operation, args = (e.headers[i],));
    futures.start();
    futures.join();

print(e.dict1)

The difference being that e in multiprocessing is not a shared object, so e does not update between all processes. In multithreading, they share the same variables, so e can be updated as expected.

Edit:

I'm unfamiliar with concurrent, but I'm sure it has a similar method to sharing types and objects like multiprocessing does shown here and objects using managers like the one here.

If speed is what you're looking for, definitely go the multiprocessing route and share your object using a manager like the one from multiprocessing.managers import BaseManager.

Hello again, thank for the answer it's working now, but i lost all the speed from the previous method ? Any guess ? — M1n1M1n1ng, Sep 29 '17 at 14:39
Yep. Threads don't offer speed improvements in python. See my answer's edit. — Treyten Carey, Sep 29 '17 at 14:43
yes speed is the only purpose, ok i will look BaseManager ! i 'll return soon to give answer on this post. Thank you for your time — M1n1M1n1ng, Sep 29 '17 at 14:47

score 1 · Answer 2 · answered Sep 29 '17 at 14:59

with the Help of @Treyten Care

Need to add to my initial post:

from multiprocessing import Process, Manager 

class exempleStackOverFlow: 
        def __init__(self):
            manager = Manager()
            self.dict1 = manager.dict()

With this i can get my dict updated. Thank

Multiprocessing and dict empty after update [python3]

2 Answers2