What do python generators really accomplish?

Question

The following paragraph is an exact quote from the book

https://www.amazon.com/Programming-Language-Explorations-Ray-Toal/dp/149873846X/ref=sr_1_fkmr1_1?s=books&ie=UTF8&qid=1541545240&sr=1-1-fkmr1&keywords=Programming+Language+Explorati+1st+Edition

Writing iterables and iterators from scratch can be a bit tedious, so Python provides generators to simplify the creation of iterable sequences. A generator results from calling a function containing a yield statement. Executing such a function does not invoke the function's body, but rather returns a generator object. Here's a generator that produces successive powers of two, up to some limit:

def powers_of_two(limit): 
   value = 1
   while value < limit: 
      yield value
      value += value

# Use the generator

for i in powers_of_two(70): 
   print(i)

1
2
4
8
16
32
64

The following 2 line function accomplishes the exact same thing without using a generator and is certainly not "a bit tedious" as the author suggests.

def alternative_powers_of_two(limit):
   for i in range(int(math.log(limit,2)+1)):
       print(2**i)

So what is the value of this generator stuff? Rather than simplifying writing iterators, it seems to do the exact opposite -- a least in the example the author uses.

In your two-line alternative, you have not written an iterator. Do you know what an iterator *is*? — user2357112, Nov 06 '18 at 23:51
You have not written an iterator anywhere here, so I'm not sure how you are concluding this isn't making it simpler. — juanpa.arrivillaga, Nov 06 '18 at 23:52
@ user2357112 I don't know exactly what an iterator is, although I would think iterating through a range would qualify. But what I did do was provide a solution to the original problem is a much more intuitive and simpler way. — redwood_gm, Nov 06 '18 at 23:57
The generator lazily produces values as they're needed. The values can be generated and consumed by whatever, whenever in the future. Your example strictly consumes values produced by the range, and without making it higher order, callers can't control how the values are used. — Carcigenicate, Nov 06 '18 at 23:58
@ Carcigenicate Are you suggesting that once the generator is available different subsets of the output can be produced by drilling down using various filters? For example, show me the powers of 2 which are also powers of 4. — redwood_gm, Nov 07 '18 at 00:24
@redwood_gm: An iterator is involved in iterating through a range, but it's not an iterator you wrote. You're using the `range` class's iterator implementation, then wondering why this seems so much easier than writing your own iterator. It's not really a surprise that using other people's work is often easier than doing it yourself. — user2357112, Nov 07 '18 at 00:28

RafazZ · Answer 1 · 2018-11-07T00:34:16.690

For starters, we need to define iterable, iterator, and generator.

An iterable in Python is an object that either has an __iter__ method, which returns an iterator, or defines a __getitem__ method, which takes a sequential index (0-based), and returns some object (or raises IndexError).
An iterator is an object that has a next (Python 2) or __next__ (Python 3) methods which return the next object in a sequence at every call or raise a StopIteration. With that said, we can say that "Every iterator is an iterable, but not every iterable is an iterator".
A generator is a "lazy iterator", which "promises" to return the values at the next call, without allocating any memory to those sequences. With that said, we can say that "Every generator is an iterator, but not every iterator is a generator".

With regard to the statement of "convenience", consider an example:

class MyAwesomeIterator(object):
  def __init__(self, limit):
    self.idx = 0
    self.limit = limit
  def __iter__(self): return self
  def next(self):
    if self.idx >= self.limit:
      raise StopIteration
    current = 2**self.idx
    self.idx += 1
    return current

As a generator the same thing would be

def MyAwesomeGenerator(limit):
  for idx in range(limit):
    yield 2**idx

Or even shorter (genexp):

MyAwesomerGenerator = (2**idx for idx in range(limit))

These definitions aren't quite accurate. An iterator is also required to have an `__iter__` method that returns `self`; iterators that do not have such a method are violating the iterator protocol, even if they can sometimes get away with it. Also, generators are a specific kind of iterator. People often mistake any lazy iterator for a generator, but only the kind of iterators created with `yield` or with genexps qualify. — user2357112, Nov 07 '18 at 00:32
And yea, the whole `lazy` evaluation is tots a longer topic to discuss :))) Thanks for pointing it out — RafazZ, Nov 07 '18 at 00:40
Can also use `yield from`: `def alt_alt_powers_of_two(limit): yield from (2*i for i in range(int(math.log(limit, 2)+1)))` — , Nov 07 '18 at 00:44
@wowserx OMG -- I've never seen constructs like that!!! I was looking for a way in my own codes to yield from an existing generator!!! COOool! — RafazZ, Nov 07 '18 at 00:47

What do python generators really accomplish?

1 Answers1