How does this loop works: for item in iter(queue.get, None)?

Question

I'm using python Queue to store items to be processed by threads. From what I've read online, putting a 'None' object in the queue and setting up the thread processing like this will make the thread stop. (which it does)

for item in iter(queue.get, None):
    #do stuff
    queue.task_done()

Now I cannot find many information online about this type of for loop. From what I've seen it just ends and will not process anything else which leaves the None object in the queue. Adding a queue.task_done() at the end doesn't work.

Can someone explain to me more about this type of for loops. How are they named and how do they work in general, or point me towards some good documentation about it since I cannot find any.

The key in this loop is an alternative signature to `iter()` - instead of calling it with an iterable, it's used as `iter(callable, sentinel)`, where it will call `callable()` and yield the result until it equals the `sentinel`, indicating the end of iteration. See [this answer](http://stackoverflow.com/a/25611913/1599111) for an easier to understand example. — Lukas Graf, Apr 20 '15 at 21:42
Note that you need to add a call to `queue.task_done()` call after the for loop to mark the sentinel task as complete, otherwise `queue.join()` will hang. — dano, Apr 20 '15 at 22:00
related: [What are the uses of iter(callable, sentinel)?](https://stackoverflow.com/q/38087427/9059420) — Darkonaut, Nov 26 '19 at 16:28

score 8 · Answer 1 · answered Apr 20 '15 at 21:42

8

This line:

for item in iter(queue.get, None):

Reads

"Keep calling queue.get, and then run the loop body, until queue.get() == None

answered Apr 20 '15 at 21:42

Eric

95,302
53
242
374

Daniel Pryden · Accepted Answer · 2015-04-20T21:44:19.167

5

See the documentation for the iter() function:

If the second argument, sentinel, is given, then o must be a callable object. The iterator created in this case will call o with no arguments for each call to its next() method; if the value returned is equal to sentinel, StopIteration will be raised, otherwise the value will be returned.

So the loop's behavior is equivalent to:

sentinel = None
while True:
  item = queue.get()
  if item == sentinel:
    break
  # do stuff
  queue.task_done()

edited Apr 20 '15 at 21:44

answered Apr 20 '15 at 21:41

Daniel Pryden

59,486
16
97
135

1

`is sentinel` or `== sentinel`? – Eric Apr 20 '15 at 21:43
@Eric: Good catch. Yes, the semantics are those of `== sentinel`. Updated. – Daniel Pryden Apr 20 '15 at 21:45

score 1 · Answer 3 · edited Jun 20 '20 at 09:12

It is a second form of iter():

iter(callable, sentinel) -> iterator
In the second form, the callable is called until it returns the sentinel.

(from help(iter)).

So each time for-statement tries to get an element from iterator returned by iter, it calls queue.get(), checks if returned not None, and passed returned value as item to for-statement-body. When somebody pushes None-terminator onto queue, queue.get() returns None causing iterator to throw StopIteration and interpreter leaves for-statement block.

score 0 · Answer 4 · answered Oct 23 '17 at 03:52

0

I think the curial part is that:

NOTE This will block when there's no element remain and no sentinel value is put.

so basically this method does not automatically run on any queue, you have to put a None value to indicate the end of process

answered Oct 23 '17 at 03:52

ospider

9,334
3
46
46

score 0 · Answer 5 · answered Nov 26 '19 at 16:21

0

I noticed that the approaches differ in a specific multiprocessing environment. Described here

answered Nov 26 '19 at 16:21

kotrfa

1,191
15
22

How does this loop works: for item in iter(queue.get, None)?

5 Answers5