Interrupting a Queue.get

Question

How can I interrupt a blocking Queue.get() in Python 3.X?

In Python 2.X setting a long timeout seems to work but the same cannot be said for Python 3.5.

Running on Windows 7, CPython 3.5.1, 64 bit both machine and Python. Seems like it does not behave the same on Ubuntu.

Answer 1

The reason it works on Python 2 is that Queue.get with a timeout on Python 2 is implemented incredibly poorly, as a polling loop with increasing sleeps between non-blocking attempts to acquire the underlying lock; Python 2 doesn't actually feature a lock primitive that supports a timed blocking acquire (which is what a Queue internal Condition variable needs, but lacks, so it uses the busy loop). When you're trying this on Python 2, all you're checking is whether the Ctrl-C is processed after one of the (short) time.sleep calls finishes, and the longest sleep in Condition is only 0.05 seconds, which is so short you probably wouldn't notice even if you hit Ctrl-C the instant a new sleep started.

Python 3 has true timed lock acquire support (thanks to narrowing the number of target OSes to those which feature a native timed mutex or semaphore of some sort). As such, you're actually blocking on the lock acquisition for the whole timeout period, not blocking for 0.05s at a time between polling attempts.

It looks like Windows allows for registering handlers for Ctrl-C that mean that Ctrl-C doesn't necessarily generate a true signal, so the lock acquisition isn't interrupted to handle it. Python is informed of the Ctrl-C when the timed lock acquisition eventually fails, so if the timeout is short, you'll eventually see the KeyboardInterrupt, but it won't be seen until the timeout lapses. Since Python 2 Condition is only sleeping 0.05 seconds at a time (or less) the Ctrl-C is always processed quickly, but Python 3 will sleep until the lock is acquired.

Ctrl-Break is guaranteed to behave as a signal, but it also can't be handled by Python properly (it just kills the process) which probably isn't what you want either.

If you want Ctrl-C to work, you're stuck polling to some extent, but at least (unlike Python 2) you can effectively poll for Ctrl-C while live blocking on the queue the rest of the time (so you're alerted to an item becoming free immediately, which is the common case).

import time
import queue

def get_timed_interruptable(q, timeout):
    stoploop = time.monotonic() + timeout - 1
    while time.monotonic() < stoploop:
        try:
            return q.get(timeout=1)  # Allow check for Ctrl-C every second
        except queue.Empty:
            pass
    # Final wait for last fraction of a second
    return q.get(timeout=max(0, stoploop + 1 - time.monotonic()))                

This blocks for a second at a time until:

1. The time remaining is less than a second (it blocks for the remaining time, then allows the Empty to propagate normally)
2. Ctrl-C was pressed during the one second interval (after the remainder of that second elapses, KeyboardInterrupt is raised)
3. An item is acquired (if Ctrl-C was pressed, it will raise at this point too)

Answer 2

As mentioned in the comment thread to the great answer @ShadowRanger provided above, here is an alternate simplified form of his function:

import queue


def get_timed_interruptable(in_queue, timeout):
    '''                                                                         
    Perform a queue.get() with a short timeout to avoid                         
    blocking SIGINT on Windows.                                                 
    '''
    while True:
        try:
            # Allow check for Ctrl-C every second                               
            return in_queue.get(timeout=min(1, timeout))
        except queue.Empty:
            if timeout < 1:
                raise
            else:
                timeout -= 1

And as @Bharel pointed out in the comments, this could run a few milliseconds longer than the absolute timeout, which may be undesirable. As such here is a version with significantly better precision:

import time
import queue


def get_timed_interruptable_precise(in_queue, timeout):
    '''                                                                         
    Perform a queue.get() with a short timeout to avoid                         
    blocking SIGINT on Windows.  Track the time closely
    for high precision on the timeout.                                                 
    '''
    timeout += time.monotonic()
    while True:
        try:
            # Allow check for Ctrl-C every second                               
            return in_queue.get(timeout=max(0, min(1, timeout - time.monotonic())))
        except queue.Empty:
            if time.monotonic() > timeout:
                raise

Answer 3

Just use get_nowait which won't block.

import time
...

while True:
  if not q.empty():
    q.get_nowait()
    break
  time.sleep(1) # optional timeout

This is obviously busy waiting, but q.get() does basically the same thing.