Confusion about iterators and iterables in Python

Question

I am currently reading in the official documentation of Python 3.5.

It states that range() is iterable, and that list() and for are iterators. [section 4.3]

However, here it states that zip() makes an iterator.

My question is that when we use this instruction:

list(zip(list1, list2))

are we using an iterator (list()) to iterate through another iterator?

`for` is not an iterator. `range()` is a *sequence*, just like `list` is, and both are iterable. You create a new iterator for an iterable to do the actual iteration. — Martijn Pieters, Nov 17 '16 at 08:41
The word iterator has actually been used with two different meanings. I recommend not fussing about the exact meanings of the words. — Alex Hall, Nov 17 '16 at 08:41
`list()` and `for` both ask an iterable for an iterator. Iterators are their own iterators. `zip()` is an iterator, not an iterable, because it can't produce new iterators (it can only be iterated over once). — Martijn Pieters, Nov 17 '16 at 08:42
@MartijnPieters Well that official Python tutorial page does say *"the for statement is such an iterator. The function list() is another"*. I guess they mean it's a thing that iterates (over the given thing). — Stefan Pochmann, Nov 17 '16 at 08:44
I think the confusion here is between the consumer (*driving* the iteration) and the producer (*can be used as*). — Martijn Pieters, Nov 17 '16 at 08:46
@ammarx: sorry about that; iterators and iterables and consumers of those can all look a bit complicated. — Martijn Pieters, Nov 17 '16 at 14:55

Martijn Pieters · Accepted Answer · 2016-11-17T15:43:45.330

The documentation is creating some confusion here, by re-using the term 'iterator'.

There are three components to the iterator protocol:

Iterables; things you can potentially iterate over and get their elements, one by one.
Iterators; things that do the iteration. Every time you want to step through all items of an iterable, you need one of these to keep track of where you are in the process. These are not re-usable; once you reach the end, that's it. For most iterables, you can create multiple indepedent iterators, each tracking position independently.
Consumers of iterators; those things that want to do something with the items.

A for loop is an example of the latter, so #3. A for loop uses the iter() function to produce an iterator (#2 above) for whatever you want to loop over, so that "whatever" must be an iterable (#1 above).

range() is an example of #1; it is iterable object. You can iterate over it multiple times, independently:

>>> r = range(5)
>>> r_iter_1 = iter(r)
>>> next(r_iter_1)
0
>>> next(r_iter_1)
1
>>> r_iter_2 = iter(r)
>>> next(r_iter_2)
0
>>> next(r_iter_1)
2

Here r_iter_1 and r_iter_2 are two separate iterators, and each time you ask for a next item they do so based on their own internal bookkeeping.

list() is an example of both an iterable (#1) and a iteration consumer (#3). If you pass another iterable (#1) to the list() call, a list object is produced containing all elements from that iterable. But list objects themselves are also iterables.

zip(), in Python 3, takes in multiple iterables (#1), and is itself an iterator (#2). zip() stores a new iterator (#2) for each of the iterables you gave it. Each time you ask zip() for the next element, zip() builds a new tuple with the next elements from each of the contained iterables:

>>> lst1, lst2 = ['foo', 'bar'], [42, 81]
>>> zipit = zip(lst1, lst2)
>>> next(zipit)
('foo', 42)
>>> next(zipit)
('bar', 81)

So in the end, list(zip(list1, list2)) uses both list1 and list2 as iterables (#1), zip() consumes those (#3) when it itself is being consumed by the outer list() call.

@Zero: by travelling by plane to another country; I jump in on the short bursts of connectivity ;-) — Martijn Pieters, Nov 17 '16 at 15:45
From your answer and @ZeroPiraeus answer, I can see that the documentation uses the term `iterator` sometimes to refer to the technical actual meaning of `iterator`, and other times to refer to the consumers of iterators. — Ammar Alyousfi, Nov 17 '16 at 15:46
But why isn't `range()` considered to be an `iterator` instead of `iterable`? Doesn't it work in the same way `zip()` works? — Ammar Alyousfi, Nov 17 '16 at 15:48
No, it is a virtual sequence. It has a length, containment tests work, you can compare two ranges and you can reverse it. You can't do that with an iterator. — Martijn Pieters, Nov 17 '16 at 15:53
Thank you for your eloborations. For the last paragraph in your answer, `zip()` is considered to be an iterator and a consumer at the same time? — Ammar Alyousfi, Nov 17 '16 at 18:19
@ammarx: yes, `zip()` is an iterator whose output is taken from iterable inputs. — Martijn Pieters, Nov 17 '16 at 19:43

score 1 · Answer 2 · edited Jun 20 '20 at 09:12

The documentation is badly worded. Here's the section you're referring to:

We say such an object is iterable, that is, suitable as a target for functions and constructs that expect something from which they can obtain successive items until the supply is exhausted. We have seen that the for statement is such an iterator. The function list() is another; it creates lists from iterables:

In this paragraph, iterator does not refer to a Python iterator object, but the general idea of "something which iterates over something". In particular, the for statement cannot be an iterator object because it isn't an object at all; it's a language construct.

To answer your specific question:

... when we use this instruction:
list(zip(list1, list2))
are we using an iterator (list()) to iterate through another iterator?

No, list() is not an iterator. It's the constructor for the list type. It can accept any iterable (including an iterator) as an argument, and uses that iterable to construct a list.

zip() is an iterator function, that is, a function which returns an iterator. In your example, the iterator it returns is passed to list(), which constructs a list object from it.

A simple way to tell whether an object is an iterator is to call next() with it, and see what happens:

>>> list1 = [1, 2, 3]
>>> list2 = [4, 5, 6]

>>> zipped = zip(list1, list2)
>>> zipped
<zip object at 0x7f27d9899688>
>>> next(zipped)
(1, 4)

In this case, the next element of zipped is returned.

>>> list3 = list(zipped)
>>> list3
[(2, 5), (3, 6)]

Notice that only the last two elements of the iterator are found in list3, because we already consumed the first one with next().

>>> next(list3)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'list' object is not an iterator

This doesn't work, because lists are not iterators.

>>> next(zipped)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

This time, although zipped is an iterator, calling next() with it raises StopIteration because it's already been exhausted to construct list3.

Confusion about iterators and iterables in Python

2 Answers2