Python threads to accumulate items to a collection

Question

I have threads to add items to let's say a global success list.

I know that thread communications should be done via Queues, but when reading the Queue documentation, there is no function to get exact count of the queue length.

Queue.qsize() Return the approximate size of the queue. Note, qsize()

0 doesn’t guarantee that a subsequent get() will not block, nor will qsize() < maxsize guarantee that put() will not block.

I have also tried len(queue) but it doesn't work:

>>> import Queue
>>> q = Queue.Queue()
>>> q.put(1)
>>> q.put(2)
>>> len(q)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: Queue instance has no attribute '__len__'

I have also read about using List but I don't have definite answer if the data will be corrupted or not.

My only option I can think of right now is to run queue.get() until queue is empty to convert to a list. Is there a better choice?

Why not subclass `Queue` and add your own `length` function or implement `__len__`? This is probably a bad idea and there's no guarantee it will be be thread safe, but it's worth a shot. — Akshat Mahajan, Apr 14 '16 at 22:01
I think when I read the documentation, it says "return the approximate size of the queue", what I need is after the thread execution block, at the end of the main thread to work out how many items in the queue. — James Lin, Apr 14 '16 at 22:43

score 1 · Accepted Answer · answered Apr 14 '16 at 22:13

The reason qsize doesn't give these guarantees is because between calling .qsize() and .get() some other thread may insert an element. Other then that you should use qsize. It will give you an exact size of the queue at the moment of calling. The word "approximate" here means that after that moment of calling the size is no longer guaranteed to be the same. But when dealing with threads accessing shared data structure you will always face the same issue, no matter what the structure is.

Python threads to accumulate items to a collection

1 Answers1