Inconsistent behavior of python generators

Question

The following python code produces [(0, 0), (0, 7)...(0, 693)] instead of the expected list of tuples combining all of the multiples of 3 and multiples of 7:

multiples_of_3 = (i*3 for i in range(100))
multiples_of_7 = (i*7 for i in range(100))
list((i,j) for i in multiples_of_3 for j in multiples_of_7)

This code fixes the problem:

list((i,j) for i in (i*3 for i in range(100)) for j in (i*7 for i in range(100)))

Questions:

1. The generator object seems to play the role of an iterator instead of providing an iterator object each time the generated list is to be enumerated. The later strategy seems to be adopted by .Net LINQ query objects. Is there an elegant way to get around this?
2. How come the second piece of code works? Shall I understand that the generator's iterator is not reset after looping through all multiples of 7?
3. Don't you think that this behavior is counter intuitive if not inconsistent?

Answer 1

A generator object is an iterator, and therefore one-shot. It's not an iterable which can produce any number of independent iterators. This behavior is not something you can change with a switch somewhere, so any work around amounts to either using an iterable (e.g. a list) instead of an generator or repeatedly constructing generators.

The second snippet does the latter. It is by definition equivalent to the loops

for i in (i*3 for i in range(100)):
    for j in (i*7 for i in range(100)):
        ...

Hopefully it isn't surprising that here, the latter generator expression is evaluated anew on each iteration of the outer loop.

Answer 2

As you discovered, the object created by a generator expression is an iterator (more precisely a generator-iterator), designed to be consumed only once. If you need a resettable generator, simply create a real generator and use it in the loops:

def multiples_of_3():               # generator
    for i in range(100):
       yield i * 3
def multiples_of_7():               # generator
    for i in range(100):
       yield i * 7
list((i,j) for i in multiples_of_3() for j in multiples_of_7())

Your second code works because the expression list of the inner loop ((i*7 ...)) is evaluated on each pass of the outer loop. This results in creating a new generator-iterator each time around, which gives you the behavior you want, but at the expense of code clarity.

To understand what is going on, remember that there is no "resetting" of an iterator when the for loop iterates over it. (This is a feature; such a reset would break iterating over a large iterator in pieces, and it would be impossible for generators.) For example:

multiples_of_2 = iter(xrange(0, 100, 2))  # iterator
for i in multiples_of_2:
    print i
# prints nothing because the iterator is spent
for i in multiples_of_2:
    print i

...as opposed to this:

multiples_of_2 = xrange(0, 100, 2)        # iterable sequence, converted to iterator
for i in multiples_of_2:
    print i
# prints again because a new iterator gets created
for i in multiples_of_2:
    print i

A generator expression is equivalent to an invoked generator and can therefore only be iterated over once.

Answer 3

The real issue as I found out is about single versus multiple pass iterables and the fact that there is currently no standard mechanism to determine if an iterable single or multi pass: See Single- vs. Multi-pass iterability

Answer 4

If you want to convert a generator expression to a multipass iterable, then it can be done in a fairly routine fashion. For example:

class MultiPass(object):
    def __init__(self, initfunc):
        self.initfunc = initfunc
    def __iter__(self):
        return self.initfunc()

multiples_of_3 = MultiPass(lambda: (i*3 for i in range(20)))
multiples_of_7 = MultiPass(lambda: (i*7 for i in range(20)))
print list((i,j) for i in multiples_of_3 for j in multiples_of_7)

From the point of view of defining the thing it's a similar amount of work to typing:

def multiples_of_3():
    return (i*3 for i in range(20))

but from the point of view of the user, they write multiples_of_3 rather than multiples_of_3(), which means the object multiples_of_3 is polymorphic with any other iterable, such as a tuple or list.

The need to type lambda: is a bit inelegant, true. I don't suppose there would be any harm in introducing "iterable comprehensions" to the language, to give you what you want while maintaining backward compatibility. But there are only so many punctuation characters, and I doubt this would be considered worth one.