7

Let's say I have two processes: a reader and a writer. How does the writer detect when the reader has finished writing values?

The multiprocessing module has a queue with a close method that seems custom-built for this purpose. But how do you detect when the queue has been closed?

This doesn't seem to work, as the writer never exits:

import multiprocessing as mp

def getter(q):
    while True:
        try:
            print(q.get())
        except Exception:
            break

def putter(q):
    q.put(1)
    q.put(2)
    q.close()

q = mp.Queue()
writer = mp.Process(target=putter, args=(q, ))
reader = mp.Process(target=getter, args=(q, ))
reader.start()
writer.start()

writer.join()
reader.join()

Should the reader-writer pair use a sentinel value to signal end of writing? Then what's the point of having the close method?

EDIT: While this question asks about the Queue module (now queue), I am asking specifically about mp.Queue and what the correct use of the .close method is.

Daniel Kats
  • 5,141
  • 15
  • 65
  • 102
  • Possible duplicate of [Communicating end of Queue](https://stackoverflow.com/questions/3605188/communicating-end-of-queue) – Turn Feb 01 '18 at 18:47
  • 1
    @Turn I'm specifically asking about `mp.Queue` which has a `.close` method. I know that this question has been asked previously about the `queue` module (formerly Queue in python 2), but both the implementation and the interfaces for these modules differ. – Daniel Kats Feb 01 '18 at 19:20
  • Fair enough, I retracted the vote. – Turn Feb 01 '18 at 19:22

1 Answers1

11

But how do you detect when the queue has been closed?

You don't. That is not the purpose of close. Calling close doesn't even guarantee that no more items will be added to the queue; the docs say

Indicate that no more data will be put on this queue by the current process.

close is intended to shut down the current process's feeder thread for that queue (or at least start shutting it down), not to communicate an end-of-queue to other processes.

If you want to signal that no more values will be written to the queue, use standard techniques like enqueueing a sentinel object, like you would with an ordinary queue.Queue.

user2357112
  • 260,549
  • 28
  • 431
  • 505