Python Multiprocessing - shared memory

Question

From the main process, I create 3 child processes and I pass an instance of a 'common' class...the same instance is passed to all 3 child processes.
This common class has a dictionary and a queue (which contains a lot of items)
These child processes retrieve 'items' from this queue
For these items, I call a REST service to get some data about the 'item'
I add this info to the "common" dictionary
There are no errors

However, when I try to access this dictionary from the main process, its empty.

I think the problem in the place of instantiation of the variable — Mohamed Abd El Raouf, Feb 12 '14 at 18:31
the behaviour is expected. See [Python: Possible to share in-memory data between 2 separate processes](http://stackoverflow.com/q/1268252/4279) — jfs, Feb 12 '14 at 20:39

score 0 · Answer 1 · answered Feb 12 '14 at 20:09

0

For sharing state between processes, the most flexible way is to use a multiprocessing.Manager. You can also use e.g. multiprocessing.Array, but that can only contain data of the same type.

answered Feb 12 '14 at 20:09

Roland Smith

42,427
3
64
94

score 0 · Answer 2 · answered Feb 17 '14 at 12:07

Thanks for your responses.

I managed to solve the problem using multiprocessing.Manager().dict()

Is this the best way....I'm not entirely sure. I think I need to read a lot more on Multiprocessing.

What I find challenging is the fact that the multiprocessing module offers a lot of functionality and as a beginner, its quite challenging to know the right 'tools' to use for the job.

I started off with threads...then moved to Processes....then used Queues....then used Managers. But as I read more on this subject, I see Python has a lot more to offer.

Python Multiprocessing - shared memory

2 Answers2