Why does my generator function doesn't loop?

Question

I'm creating a python generator to loop over a sentence. "this is a test" should return

this
is
a
test

Question1: What's the problem with the implementation below? It only return "this" and doesn't loop. how to fix it?

def my_sentense_with_generator(sentence):
    index = 0
    words = sentence.split()
    current = index
    yield words[current]
    index +=1

for i in my_sentense_with_generator('this is a test'):
   print(i) 
>> this

Question2 : Another way of implementation is below. It works. But i'm confused about the purpose of using 'for' here. I was taught that in one way, generator is used in lieu of "for loop" so that python doesn't have to build up the the whole list upfront, so it takes much less memory and time link. But in this solution, it uses for loop to construct a generator.. does it defeat the purpose of generator??

def my_sentense_with_generator(sentence):
    for w in sentence.split():
        yield w

Does this answer your question? [What does the "yield" keyword do?](https://stackoverflow.com/questions/231767/what-does-the-yield-keyword-do) — wovano, Nov 07 '21 at 10:47
*"Question 1... question 2"*: On Stack Overflow you should ask one unique question. Multiple questions are reason for closing. — trincot, Nov 07 '21 at 11:15

mozway · Answer 1 · 2021-11-07T10:35:49.473

The purpose of a generator is not to avoid defining a loop, it is to generate the elements only when they are needed (and not when it is constructed)

In your 1st example, you need a loop in the generator as well. Otherwise the generator is only able to generate a single element, then it is exhausted.

NB. In the generator below, the str.split creates a list, so there is no memory benefit in using a generator. This could be replaced by an iterator iter(sentence.split())

def my_sentence_with_generator(sentence):
    words = sentence.split()
    for word in words:
        yield word

for i in my_sentence_with_generator('this is a test'):
    print(i)

output:

this
is
a
test

The loop in the generator defines the elements of the generator, if will pause at a yield until something requests an element of the generator. So you also need one loop outside the generator to request the elements.

Example of a partial collection of the elements:

g = my_sentence_with_generator('this is a test')

next(g), next(g)

output: ('this', 'is')

example of the utility of a generator:

def count():
    '''this generator can yield 1 trillion numbers'''
    for i in range(1_000_000_000_000_000):
        yield i
     

# we instanciate the generator   
c = count()

# we collect only 3 elements, this consumes very little memory
next(c), next(c), next(c)

please read through my question. this is my question no.2 i already gave this solution — somniumm, Nov 07 '21 at 09:41
@somniumm I was adding more details, please have a look if this answers your question — mozway, Nov 07 '21 at 09:42
i understand question2 now. I'm confused where i did wrong in the first implementation. or is there a way to fix it rather than using for loop — somniumm, Nov 07 '21 at 10:01
What is the purpose of the first implementation? Because you're starting with a `str.split`, this creates a list, so there is absolutely no advantage in doing a generator here. The list will be constructed from the beginning. If the purpose is to have an iterator, you could simply do `iter(sentence.split())` — mozway, Nov 07 '21 at 10:33

score 0 · Answer 2 · answered Nov 07 '21 at 10:04

str.split returns a list, so you're not going to avoid creating a list if you call that within your generator. And you'll either need to keep the original string in memory until you're done with the results, or create twice as many strings as you'd need otherwise if you don't want to have that list, otherwise it'll be impossible to figure out what next to yield.

As an example, this is what a version of str.split might look like as a generator:

def split_string(sentence, sep=None):
    # Creates a list with a maximum of two elements: 
    split_list = sentence.split(sep, maxsplit=1)
    while split_list:
        yield split_list[0]
        if len(split_list) == 1:
            # No more separators to be found in the rest of the string, so we return
            return
        split_list = split_list[1].split(sep, maxsplit=1)

This creates a lot of short-lived lists, but they will never have more than 2 elements each. Not very practical, it's likely to be much less performant than just calling str.split once, but hopefully it gives you a better understanding of how generators work.

Why does my generator function doesn't loop?

2 Answers2

example of the utility of a generator: