3

The gist of my question is what happens to a multiprocessing queue when the parent (a daemon in this circumstance) is killed.

I have a daemon that runs in the background which queues up jobs for child processes:

class manager(Daemon):
    def run(self):
        someQueue = MP.Queue()

        someChild = MP.Process(target=someCode, args=(someArgs))
        someChild.start()
        ...

If the manager is killed (assuming it wasn't trying to use someQueue and therefore corrupted it as mentioned in the documentation), is there anyway to recover the data in the Queue?

Two theoretical solutions I see are cleaning up the someQueue in someChild before exiting this child process. Also dumping the queues so that I could restore the state of the queues when the manager exited would also solve my problem. However, before implementing either it would be nice to get nudged in the right direction.

Thanks,

Aaron Robinson
  • 406
  • 1
  • 6
  • 13

1 Answers1

1

It sounds like you want persistent/reliable queuing. I believe the multiprocessing.Queue class is implemented with pipes (just like you would get with a popen() call), so the data is relatively transient and you'd probably have to do some OS-level trickery to grab the contents. You might look into writing your own persistent queue class that uses a filesystem file (assuming your OS and filesystem support locking) to store the queue contents. Then you can provide all of the analysis tools you desire to inspect the queue and recover unprocessed data.

ColoradoEric
  • 61
  • 4
  • I don't see how this answers the question. Multiprocessing has a Pipe as well as a Queue, so one would assume that both the pipe and Queue would work similar in this circumstance (as long as in both cases your avoiding the fore mentioned corrupted cases in the MP documentation). – Aaron Robinson Jan 31 '12 at 22:27
  • I felt confident in believing that there was OS-level trickery to get the data of these queues but you don't mention any method to attempt to do so? I am using python so why write a persistent queue class when surely there has to be some module that can help with these circumstances. – Aaron Robinson Jan 31 '12 at 22:29
  • *(...) `Queue()` is built on top of `Pipe()`.* - [Python multiprocessing - Pipe vs Queue](http://stackoverflow.com/q/8463008/95735) – Piotr Dobrogost Feb 01 '12 at 09:47