0

I've written a simple generator function that takes a list that could have sub-lists and tries to flatten the list:

so [1, [2, 3], 4, [5, [6, 7], 8]] should produce 1,2,3,4,5,6,7,8

If I just want to print out the values (not a generator) it looks like this and this works:

#  Code A
def flatten_list_of_lists(my_list):
    for element in my_list:
        if isinstance(element, list):
            flatten_list_of_lists(element)
        else:
            print(element)

my_list = [1, [2, 3], 4, [5, [6, 7], 8]]
flatten_list_of_lists(my_list)

And that prints out 1,2,3,4,5,6,7,8 as expected

However, when I change the code to this:

#  Code B
def flatten_list_of_lists(my_list):
    for element in my_list:
        if isinstance(element, list):
            flatten_list_of_lists(element)
        else:
            yield element

for i in flatten_list_of_lists(my_list):
    print(i)

which is just switching over the print to a yield, the program just prints out 1,4.

I'll paste code below that actually works. But I'm wondering why the previous code doesnt work? If Code A 'prints' out the numbers correctly, why wouldnt Code B 'yield' the numbers correctly?

Seems like I have a fundamental misunderstanding of how generators work with recursion.

This code actually works:

#  Code C
def flatten_list_of_lists_v2(my_list):
    for element in my_list:
        if isinstance(element, list):
            for sub_element in flatten_list_of_lists_v2(element):
                yield sub_element
        else:
            yield element

l = []
for element in flatten_list_of_lists_v2(my_list):
    print(element)

And that prints out 1,2,3,4,5,6,7,8

Just a little background, I just finished watching this video: https://www.youtube.com/watch?v=LelQTPiH3f4

and in there he explains when you're designing your generators, just put a print where you want to yield and see if you get the right results and then just switch the print to a yield. So I guess his advice doesnt work in all circumstances, I just want to understand why.

Benjamin Kozuch
  • 150
  • 1
  • 8
  • 5
    in `code A`, change `flatten_list_of_lists(element)` to `yield from flatten_list_of_lists(element)` and `print(element)` to `yield element` – Mulan Apr 05 '19 at 14:32
  • 2
    In code B, `flatten_list_of_lists(element)` is a statement that returns a generator but nothing is done with this object so the code continue until it reaches the next `yield` statement. – Jacques Gaudin Apr 05 '19 at 14:34
  • Similar in nature to https://stackoverflow.com/q/31221826/1126841; you are ignoring the return value of (or rather, values that can be extracted from) the recursive call to `flatten_list_of_lists_v2`. You would have the same problem in your first version if you were *returning* values instead of *printing* them. – chepner Apr 05 '19 at 14:35
  • 1
    The video is giving terrible advice. Printing a value is very different from either returning or yielding a value. – chepner Apr 05 '19 at 14:36
  • Compare `x = (lambda x: x)(3)` with `x = print(3)` (which assumes you are not using Python 2's `print` statement). – chepner Apr 05 '19 at 14:39

1 Answers1

6

A simple mistake -

def flatten_list_of_lists(my_list):
    for element in my_list:
        if isinstance(element, list):
            # add yield from
            yield from flatten_list_of_lists(element)
        else:
            # yield, not print
            yield element

my_list =  [1, [2, 3], 4, [5, [6, 7], 8]]

for e in flatten_list_of_lists(my_list):
  print(e)

Output

1
2
3
4
5
6
7
8
Mulan
  • 129,518
  • 31
  • 228
  • 259
  • But why doesnt my code work...I am calling flatten_list_of_lists with a sublist so that call to that should also yield the values within that sublist – Benjamin Kozuch Apr 05 '19 at 14:36
  • 1
    because `flatten_list_from_lists` returns a generator and you didn't _do_ anything with it! It's like saying (1 + 1) but not assigning it to a variable, or printing it... it evaluates to `2` but then vanishes – Mulan Apr 05 '19 at 14:37
  • 1
    Your first version works because it isn't *returning anything*; it *prints* values. – chepner Apr 05 '19 at 14:38
  • Thank you, I think I understand now. Basically, my loop that is calling this generator is only getting the 'first generator' returned, and not any future generators which are created within the generator. – Benjamin Kozuch Apr 05 '19 at 14:48
  • Also thank you, I never heard of 'yield from' until now. Heres something on it if anyone is interested. https://utcc.utoronto.ca/~cks/space/blog/python/YieldFromAndGeneratorFunctions – Benjamin Kozuch Apr 05 '19 at 14:49
  • @BenjaminKozuch you're welcome. Intuitively, you can think of `yield from` as a way of one generator passing its yield job onto another generator - a sort of delegation. And in the case of a _recursive_ generator, it delegates that job to itself! A glance at the program above shows that only non-list `element`s will be yielded as there is no other way for data to come out of the generator - other statements are `for`, `if`, and `else`, which don't "output" anything. – Mulan Apr 05 '19 at 14:52
  • Under the hood, is 'yield from' doing the same thing that 'Code C' (in my original question) is doing? – Benjamin Kozuch Apr 05 '19 at 15:02
  • @BenjaminKozuch yup, precisely - in this code, you have `for sub_element in ...` which actually _does_ something with the generator `flatten_list_of_lists(element)`, which is why you see it producing the intended effect. – Mulan Apr 05 '19 at 15:15