3

How can I check if a Queue is empty? Running below code :

from multiprocessing import Queue

q = Queue()
print(q.empty())

q.put("a")
print(q.empty())

renders:

True
True

Shouldnt q.empty() return False after an element has been added?

blue-sky
  • 51,962
  • 152
  • 427
  • 752
  • 2
    I'm guessing `put` is asynchronous, so you don't have any guarantees that it will have completed by the time the next line executes. – Carcigenicate Aug 14 '20 at 21:48
  • 1
    its asynchronous, let it sleep for 2 seconds then check `q.empty()` again – M Z Aug 14 '20 at 21:48
  • @MZ yes, using time.sleep(1) after adding the element returns False on empty check. Will I post a short answer or will you ? – blue-sky Aug 14 '20 at 21:52

3 Answers3

1

Two possible solutions:

  1. Use queue.Queue instead of multiprocessing.Queue: Some differences in doc between them:

They differ in that Queue lacks the task_done() and join() methods introduced into Python 2.5's queue.Queue class.

from queue import Queue
q = Queue()
print(q.empty())
q.put("a")
print(q.empty())
  1. Or just like @Hans Musgrave said in comment.time.sleep(1) is not guaranteed.You could use time.sleep(0) instead:
from multiprocessing import Queue
import time
q = Queue()
print(q.empty())
time.sleep(0)
q.put("a")
print(q.empty())

All of them give me:

True
False

About why sleep(0), A value of zero causes the thread to relinquish the remainder of its time slice to any other thread that is ready to run. If there are no other threads ready to run, the function returns immediately, and the thread continues execution cpu gives up resources, to execute q.emety() task.That's what I think.

The stable method is to use queue.Queue or queues.SimpleQueue.

jizhihaoSAMA
  • 12,336
  • 9
  • 27
  • 49
  • is time.sleep(0) not redundant? What's the point of sleeping function for 0 seconds? Isnt time.sleep(0) the same as not calling sleep function at all? – blue-sky Aug 15 '20 at 17:42
  • @blue-sky https://stackoverflow.com/questions/7273474/behavior-of-pythons-time-sleep0-under-linux-does-it-cause-a-context-switch and https://stackoverflow.com/questions/3727420/significance-of-sleep0. – jizhihaoSAMA Aug 16 '20 at 01:21
  • How does `time.sleep(0)` ensure that `q.empty()` will always return the correct value? It merely causes a context switch, and even that is not guaranteed according to the links you shared. – Anmol Singh Jaggi Aug 25 '20 at 15:49
1

The officially recommended solution is to use Manager.Queue.

Quoting from Python docs :

Note:
When an object is put on a queue, the object is pickled and a background thread later flushes the pickled data to an underlying pipe. This has some consequences which are a little surprising, but should not cause any practical difficulties – if they really bother you then you can instead use a queue created with a manager.
1. After putting an object on an empty queue there may be an infinitesimal delay before the queue’s empty() method returns False and get_nowait() can return without raising Queue.Empty.
2. If multiple processes are enqueuing objects, it is possible for the objects to be received at the other end out-of-order. However, objects enqueued by the same process will always be in the expected order with respect to each other.

Anmol Singh Jaggi
  • 8,376
  • 4
  • 36
  • 77
0

The put method is asynchronous, therefore the queue can be still empty when checking for elements. Adding time.sleep(1) after put and empty() returns False.

blue-sky
  • 51,962
  • 152
  • 427
  • 752