Python’s Iterators and Generators: Still Not Fully Understood

Question

import random

my_list = [random.randint(3, 100) for i in range(4)]

def is_even(some_list):
    for i in some_list:
        if i % 2 == 0:
            yield True
        else:
            yield False

print(my_list)
print(list(is_even(my_list)))

>>> [94, 53, 27, 42]
>>> [True, False, False, True]

It seems that I still do not fully understand the concept. On each iteration, the is_even function yields True or False accordingly. I do not see how those “yields” are accumulated. (Or, I do not see how they are appended to the final list at the end of each iteration. For example, in the given example, the first iteration yields True, then the second iteration starts. Where is that first True value kept?) What exactly is happening there?

I think you will fin your answer here: http://stackoverflow.com/questions/231767/what-does-the-yield-keyword-do-in-python — Bridgekeeper, Nov 25 '15 at 11:22

user4815162342 · Accepted Answer · 2015-11-25T13:51:52.387

To answer your question: the accumulation magic happens inside the list constructor - in fact, accumulation of items into a list is its very job. When you write:

print(list(is_even(my_list)))

The object returned by is_even(...) has a next() method that provides the next value yielded by the generator, or raises an exception when no more values are available. The list function is the one that accumulates those values into a list, which is finally returned.

You can imagine list being a function defined like this:

def list(iter):
    accum = []
    # the following loop is an approximate expansion of
    # for item in iter: accum.append(item)
    while 1:
        try:
            # get the next yielded value from generator
            item = iter.next()
        except StopIteration:
            # no more values
            break
        accum.append(item)
    return accum

In your case, where list is given a generator, the for loop is exhausting the generator and accumulating individual elements in a temporary list kept under the hood. It is this temporary list, accum in the above implementation, that holds that first True value, and goes on to store the subsequent ones. Once the generator is exhausted, the whole accumulated list is returned to the caller.

Thank you for your explanation. Now I see how `list` works. (BTW, you said “you can think of how `list` works,” is it how it really works, or I can simply imagine that way?) My main issue is, how the values are accumulated inside `is_even` function. From what you say, I understand that it generates a temporary list to hold the yielded values, and the list construct iterates over that temporary list, and copies the values into itself; is that correct? — blackened, Nov 25 '15 at 13:29
@blackened The values are not accumulated inside `is_even`, they are provided (yielded) by `is_even`, one by one. It is the caller that does the accumulation in the manner of its own choosing - `list` accumulates into a list, `tuple` into a tuple, etc. In other words, there is only one list that holds the values, and that list is returned by the `list` constructor. (The `list` constructor really works like that, except it's written in C for efficiency, and uses the `extend` method to fill the new list.) I've now amended the answer to show how obtaining individual values works in more detail. — user4815162342, Nov 25 '15 at 13:54

score 0 · Answer 2 · answered Nov 25 '15 at 11:27

I think I understand what you mean:

Each time is_even is run, it checks every number in the list and returns True or false (as i'm sure you're aware). When you use 'print(list(is_even(my_list)))' the process is very simple - you are creating a list of every result returned by the function 'is_even', in the same order the for loop processed them before hand.

Hope this has helped.

gerosalesc · Answer 3 · 2015-11-25T11:39:10.577

You can understand the iterators as a list which has all its elements unfilled until you ask for them. If you instance a list with an iterator as a parameter, like you are doing in this instruction list(is_even(my_list)) you will get all of its elements out to memory immediately, which is not what we want when using iterators because is more efficient to have only the last element to check and not the whole thing, right? so try to iterate over it once at a time either like this:

for item in iterator:

Or using next() method.

score 0 · Answer 4 · answered Nov 25 '15 at 11:50

Following your example from start:

my_list = [random.randint(3, 100) for i in range(4)]

This is a nice example of list comprehension. It is just a more concise way of writing something like:

my_list = []
for i in range(4):
    my_list.append(random.randint(3, 100))

Next, you define a generator function:

def is_even(some_list):
    for i in some_list:
        if i % 2 == 0:
            yield True
        else:
            yield False

A generator function is able to behave like an iterator. Values are "yielded" on demand. You can picture it as a function that streams results.

print(list(is_even(my_list)))

A few things are happening at this point, let's split them to get a better overview of what is going on:

>>> g = is_even(my_list)

is_even(my_list) doesn't return a list, as you may expect, but a generator:

>>> g
... <generator object is_even at ...>
>>> type(g)
... generator

The list construct accepts an iterable as argument, an a generator is iterable. Your yielded values are actually accumulated at this step, when the actual list is constructed:

>>> l = list(g)
>>> print(l)
... [True, False, False, True]

Python’s Iterators and Generators: Still Not Fully Understood

4 Answers4