Python--Getting Queue.Empty exception from a nonempty multiprocessing.Queue

Question

I've having the opposite problem of many Python users--my program is using too little CPU. I already got help in switching to multiprocessing to utilize all four of my work computer's cores, and I have seen real performance improvement as a result. But the improvement is somewhat unreliable. The CPU usage of my program seems to deteriorate as it continues to run--even with six processes running. After adding some debug messages, I discovered this was because some of the processes I was spawning (which are all supposed to run until completion) were dying prematurely. The main body of the method the processes run is a while True loop, and the only way out is this block:

try:
    f = filequeue.get(False)
except Empty:
    print "Done"
    return

filequeue is populated before the creation of the subprocesses, so it definitely isn't actually empty. All the processes should exit at roughly the same time once it actually is empty. I tried adding a nonzero timeout (0.05) parameter to the Queue.get call, but this didn't fix the problem. Why could I be getting a Queue.empty exception from a nonempty Queue?

What happens if you do `filequeue.get(True)` instead of `filequeue.get(False)`? — NPE, Aug 02 '11 at 18:17
It is a multiprocessing.Queue, since I am using multiprocessing. If I did filequeue.get(True), of course I don't get any Queue.Empty exceptions, but all the processess would block forever once filequeue has been depleted, so everything stops when the main process tries to join the spawned ones. — dpitch40, Aug 02 '11 at 18:33
To work around this, the main process could add a special "sentinel" object at the end of each queue. The workers would terminate upon seeing this special object (instead of relying on the emptiness of the queue). — NPE, Aug 02 '11 at 18:35

score 3 · Accepted Answer · answered Aug 02 '11 at 18:58

I suggest using filequeue.get(True) instead of filequeue.get(False). This will cause the queue to block until there are more elements.

It will, however, block forever after the final element has been processed.

To work around this, the main process could add a special "sentinel" object at the end of each queue. The workers would terminate upon seeing this special object (instead of relying on the emptiness of the queue).

score 2 · Answer 2 · answered Jun 23 '13 at 15:35

I had a similar problem and found out from experimentation, not reading the docs, that even when the queue is non-empty, get(False) can still spuriously throw Empty. In my use case, the workers have to exit when they run out of work in the Queue, so get(True) is a non-option.

My solution was this: I found that if in the "except Empty:" block, I check that the Queue is indeed empty(), it works -- empty() will not return True unless the Queue is really empty.

I was using Python 2.7.

Python--Getting Queue.Empty exception from a nonempty multiprocessing.Queue

2 Answers2