Python nested generators not working

Question

I am practising with generators and I wonder why the following code does not print 16 pairs but only 4.

def range_generator_function(my_range):
    for i in my_range:
        yield i

gen1=range_generator_function(range(1,5))
gen2=range_generator_function(range(1,5))

def pairs_generator_function(gen1,gen2):
    for it1 in gen1:
        for it2 in gen2:
            yield [it1,it2]

my_gen = pairs_generator_function(gen1,gen2)

for it in my_gen:
    print(it)

The output is

[1, 1]
[1, 2]
[1, 3]
[1, 4]

While the output I expect is

[1, 1]
[1, 2]
[1, 3]
[1, 4]
[2, 1]
[2, 2]
[2, 3]
[2, 4]
[3, 1]
[3, 2]
[3, 3]
[3, 4]
[4, 1]
[4, 2]
[4, 3]
[4, 4]

Generators *are single-pass iterators*. – juanpa.arrivillaga Mar 09 '18 at 23:18 — juanpa.arrivillaga, Mar 09 '18 at 23:18

wim · Accepted Answer · 2018-03-09T23:15:47.170

5

The actual output is correct. Your gen2 instance is completely exhausted by the first inner loop:

def pairs_generator_function(gen1,gen2):
    for it1 in gen1:
        for it2 in gen2:  # <--- this consumes gen2
            yield [it1,it2]

On the subsequent iterations, iterating gen2 again is empty.

edited Mar 09 '18 at 23:15

answered Mar 09 '18 at 23:15

wim

338,267
99
616
750

How can I make it usable again? – Nisba Mar 09 '18 at 23:15
1

You can not. Generators are one-shot. You will have to save the values during the first iteration, or create a new generator. – wim Mar 09 '18 at 23:16
Related: https://stackoverflow.com/questions/1271320/resetting-generator-object-in-python – Robᵩ Mar 09 '18 at 23:18

score 2 · Answer 2 · answered Mar 09 '18 at 23:16

2

You exhausted gen2 after the first time through the inner loop. You no longer have anything in that stream, so the other three values of it1 have nothing with which to pair. You need to restart gen2 every time through. Consider itertools.tee for cloning more copies.

answered Mar 09 '18 at 23:16

Prune

76,765
14
60
81

I don't see this as complex, but it is a minor shame to have to go through gen1 an extra time. However, it does leave the solution encapsulated. – Prune Mar 10 '18 at 01:53

Olivier Melançon · Answer 3 · 2018-04-02T19:49:53.750

1

The output you get is due to the fact that on the second iteration, the generator gen2 has been exhausted.

You can either make a list out of it to store its output prior to looping or copy it with itertools.tee on every iteration. The former will not be able to handle infinite generators, but before implementing the later let me point out that you are actually reimplementing itertools.product.

from itertools import product

my_gen = product(gen1, gen2)

for it in my_gen:
    print(it)

edited Apr 02 '18 at 19:49

answered Mar 09 '18 at 23:27

Olivier Melançon

21,584
4
41
73

this is really interesting! – Nisba Mar 09 '18 at 23:30
@Nisba I am not sure someone showed code to fix your issue, so I added it to my answer. – Olivier Melançon Mar 09 '18 at 23:44

score 0 · Answer 4 · answered Mar 09 '18 at 23:20

I modified the code so I get the wanted output, now the generator is recreated every time inside the outer loop.

def range_generator_function(my_range):
    for i in my_range:
        yield i

def pairs_generator_function():
    gen1=range_generator_function(range(1,5))
    for it1 in gen1:
        gen2=range_generator_function(range(1,5))
        for it2 in gen2:
            yield [it1,it2]

my_gen = pairs_generator_function()

for it in my_gen:
    print(it)

Ajax1234 · Answer 5 · 2018-03-09T23:26:48.247

As @wim pointed out, your generator is entirely exhausted after the first instance of iteration. However, to prevent this, cast gen1 and gen2 as lists when passing them to pairs_generator_function. However, you can use itertools.tee to store two copies of the original generator: one to pass to the function as a list, and the other for future use:

import itertools

def range_generator_function(my_range):
   for i in my_range:
     yield i

gen1=range_generator_function(range(1,5))
gen2=range_generator_function(range(1,5))

def pairs_generator_function(gen1,gen2):
  for it1 in gen1:
    for it2 in gen2:
        yield [it1,it2]

gen1, gen1_l = itertools.tee(gen1)
gen2, gen2_l = itertools.tee(gen2)
print(list(pairs_generator_function(list(gen1_l), list(gen2_l))))

Output:

[[1, 1], [1, 2], [1, 3], [1, 4], [2, 1], [2, 2], [2, 3], [2, 4], [3, 1], [3, 2], [3, 3], [3, 4], [4, 1], [4, 2], [4, 3], [4, 4]]

Notice, however, that gen1 and gen2` still point to the items in memory:

>>next(gen1)
1
>>next(gen2)
1

Python nested generators not working

5 Answers5