Clear all items from the queue

Question

How can I clear a queue. For example I have datas in a queue, but for some reason I don't need the existing data, and just want to clear the queue.

Is there any way? Will this work:

oldQueue = Queue.Queue()

If you read the documentation, it refers you to the Queue module's [latest source](http://docs.python.org/library/queue.html?highlight=queue#Queue.Queue), and there you could see the that the internal representation of a FIFO queue is a `deque`. In the documentation for `deque` you'd notice it has a `clear()` method, which is what you want. You'd probably also want to wrap that call with a `q.mutex.acquire()` and `q.mutex.release()` as the rest of the code does around such operations to make it thread safe. — martineau, Jun 29 '11 at 09:16

Rob Cowie · Accepted Answer · 2011-08-26T10:23:47.450

126

q = Queue.Queue()
q.queue.clear()

EDIT I omitted the issue of thread safety for clarity and brevity, but @Dan D is quite correct, the following is better.

q = Queue.Queue()
with q.mutex:
    q.queue.clear()

edited Aug 26 '11 at 10:23

answered Jun 29 '11 at 08:53

Rob Cowie

22,259
6
62
56

23

if you do `with q.mutex: q.queue.clear()`, this operation would be thread safe. – Dan D. Jun 29 '11 at 09:01
8

@DanD. please correct me if I'm wrong but this actually seems to be a **bad** idea to me. `.mutex` and `.queue` [seems to be undocumented](https://docs.python.org/2/library/queue.html) and if I'm not mistaken, using `q.mutex` would **dead-lock** if you do anything like even `q.get_nowait` inside its block. – n611x007 Sep 01 '14 at 12:07
1

@Jean-BernardJansen: It's not automatically threadsafe if you're screwing with its internals directly, and `q.queue.clear()` is screwing with the internals directly. – user2357112 Jul 25 '16 at 06:16
3

n611x007 and user2357112 are correct. I would exepect an underscore before undocumented/private members, misleading me in believing clear() was in the safe API. Clearing a queue looks like a bad idea anyway, better to read it and drop uneeded items. – Jean-Bernard Jansen Jul 25 '16 at 10:01
2

Note, as stated below in V.E.O's answer, simply clearing is not enough since even though the queue will be empty, the corresponding tasks haven't been marked as done. This may cause your code to lock up. – Chrigi Sep 17 '16 at 17:35
3

I used the approach below as well, with q.mutex: size = len(q.queue) q.queue.clear() q.unfinished_tasks -= size that sorted the issue for the join() blocking – shipperizer Jun 29 '18 at 07:35
that sorted the issue for the join() blocking, also it is an order of 400 times better in terms of time – shipperizer Jun 29 '18 at 07:40
This solution is not effective if q.join is used. This doesn't reset the unfinished job's count so they can cause unexpected side effects. (ie not being able to join) – Gerzson Oct 25 '21 at 13:58

score 47 · Answer 2 · edited Nov 06 '22 at 19:38

47

You just can not clear the queue, because every put also add the unfinished_tasks member. The join method depends on this value. And all_tasks_done needs to be notified also.

with q.mutex:
    q.queue.clear()
    q.all_tasks_done.notify_all()
    q.unfinished_tasks = 0

or in decent way, use get and task_done pair to safely clear the tasks.

    while not q.empty():
        try:
            q.get(block=False)
        except Empty:
            continue
        q.task_done()

or just create a new Queue and delete old one.

edited Nov 06 '22 at 19:38

user124114

8,372
11
41
63

answered Sep 18 '13 at 13:11

V.E.O

875
8
11

Use caution with that method, too. The [docs](https://docs.python.org/2/library/queue.html#queue-objects) note, "if empty() returns False it doesn’t guarantee that a subsequent call to get() will not block." – Lack Apr 21 '14 at 23:23
@Lack Yes, fix it by non-blocking get – V.E.O Apr 22 '14 at 10:30
3

the second way (the "decent" one) seems to be the safest and more elegant way, and uses only the documented, public API. Can anyone confirm? – MestreLion Nov 22 '14 at 11:33
It seems these would have slightly different behaviors on a concurrent put. The first will block the concurrent put, while the second will catch and remove the concurrent put on the next loop. Right? – Michael Delgado Nov 04 '15 at 22:05
The decent way has no guarantee that it will end at all. Not to mention the loop overhead. I'd stick with the first way. – freakish Dec 17 '15 at 11:08
@V.E.O you saved me a lot of headache. The accepted solution is not effective if the queue uses join(). The unfinished tasks count is not reset. – Gerzson Oct 25 '21 at 14:00

score 12 · Answer 3 · answered Aug 08 '15 at 10:37

12

This seems to do it pretty well for me. I welcome comments/additions in case I missed anything important.

class Queue(queue.Queue):
  '''
  A custom queue subclass that provides a :meth:`clear` method.
  '''

  def clear(self):
    '''
    Clears all items from the queue.
    '''

    with self.mutex:
      unfinished = self.unfinished_tasks - len(self.queue)
      if unfinished <= 0:
        if unfinished < 0:
          raise ValueError('task_done() called too many times')
        self.all_tasks_done.notify_all()
      self.unfinished_tasks = unfinished
      self.queue.clear()
      self.not_full.notify_all()

answered Aug 08 '15 at 10:37

Niklas R

16,299
28
108
203

2

This seems to be the most efficient (call to `.clear()`) and correct (notifications that include currently processed items) way to do that. – freakish Dec 17 '15 at 11:11
Would a multiprocessing implementation resemble this? – Mat90 Oct 29 '21 at 10:06

Clear all items from the queue

3 Answers3

Linked