0

I have a Python application where I create several objects that spawn one or more multiprocessing.Process objects. I want to create an object that these processes can access which will allow them to retrieve and save key/value pairs from a file. I also want the ability for a processes to be able to access key/value pairs that other processes have saved. The application is running in an environment where I only want writes to occur at specific times which are controlled by the parent process.

My initial idea was to create a singleton wrapper containing a dictionary which would load values upon instantiation and write to a file when directed to do so. I quickly found out that this just lead to a singleton that was created for every process which had no knowledge of any value generated since the last time any process saved values to a file.

Is there a way to implement the dictionary in the singleton that would be shared between processes without having to explicitly pass the dictionary into the constructor of the multiprocessing.Process object? I'm also open to alternative solutions (i.e. not use a singleton).

harumph
  • 195
  • 1
  • 11
  • Straight for the doc: [Sharing Objects Between Processes](https://docs.python.org/2/library/multiprocessing.html#exchanging-objects-between-processes) – aruisdante Oct 10 '14 at 16:51
  • [`multiprocessing.Manager().dict`](https://docs.python.org/2/library/multiprocessing.html#multiprocessing.managers.SyncManager.dict) – roippi Oct 10 '14 at 16:52
  • @aruisdante - So you're saying I can't do what I want? – harumph Oct 10 '14 at 16:59
  • Use `multiprocessing.Manager().dict`. That sounds like exactly what you want. – dano Oct 10 '14 at 17:02
  • @dano - But doesn't that require that I pass it to each and every Process constructor that I want to be able to have access it? – harumph Oct 10 '14 at 17:04
  • Yes. You're going to have to do that. There's no safe way to make a singleton (or any object really) that's magically interprocess safe. There needs to be some kind of management framework somewhere to control read/write access. But it doesn't sound like that should be an unreasonable constraint since you talk about having a 'parent' process. Failing that, you could fall back to using file IO (having processes lock a file, write to it, unlock, then other process locks, reads, unlocks) but that can be a good bit slower. – aruisdante Oct 10 '14 at 17:08
  • To put it another way, without having the object explicitly passed via the parent process, how does python know what process gets the 'master' copy of the object (since the object has to live in some processes's space somewhere)? How would it be able to tell the difference between a 'child' process that gets a reference to this 'master' object and a new parent process that's going to have its own copy of that 'master' object? – aruisdante Oct 10 '14 at 17:13
  • @aruisdante - Thanks, I was hoping there was something to the Manager.* data structures where, after first being created in a process, would implicitly be shared among other processes forked later. – harumph Oct 10 '14 at 17:20
  • If you're using Linux, you should be able to just declare the `Manager().dict` in the global scope of the main process, and it'll get inherited by the children. That won't work on Windows, though, I don't think. – dano Oct 10 '14 at 17:20

0 Answers0