Do generators simply make new objects with iter and next functions?

Question

I tried to search for this answer on my own, but there was too much noise.

Are generators in python just a convenience wrapper for the user to make an iterator object?

When you define the generator:

def test():
  x = 0
  while True:
    x += 1
    yield x

is python simply making a new object, adding the __iter__ method, then putting the rest of the code into the next function?

class Test(object):

  def __init__(self):
    self.x = 0

  def __iter__(self):
    return self

  def next(self):
    self.x += 1
    return self.x

score 2 · Answer 1 · answered Sep 16 '16 at 00:03

2

No -- Generators also provide other methods (.send, .throw, etc) and can be used for more purposes than simply making iterators (e.g. coroutines).

Indeed, generators are an entirely different beast and a core language feature. It'd be very hard (possibly impossible) to create one in vanilla python if they weren't baked into the language.

With that said, one application of generators is to provide an easy syntax for creating an iterator :-).

answered Sep 16 '16 at 00:03

mgilson

300,191
65
633
696

Thanks. I just wasn't sure if both objects boiled down to the same bytecode in the end. – Esteban Sep 16 '16 at 03:26

score 2 · Accepted Answer · answered Sep 16 '16 at 00:10

Nope. Like so:

>>> def test():
...    x = 0
...    while True:
...       x += 1
...       yield x
...
>>> type(test)
<type 'function'>

So what it returns is a function object. Details from there get hairy; the short course is that the code object belonging to the function (test.func_code) is marked as a generator by one of the flags in test.func_code.co_flags.

You can disassemble the bytecode for test to see that it's just like any other function otherwise, apart from that a generator function always contains a YIELD_VALUE opcode:

>>> import dis
>>> dis.dis(test)
  2           0 LOAD_CONST               1 (0)
              3 STORE_FAST               0 (x)

  3           6 SETUP_LOOP              25 (to 34)
        >>    9 LOAD_GLOBAL              0 (True)
             12 POP_JUMP_IF_FALSE       33

  4          15 LOAD_FAST                0 (x)
             18 LOAD_CONST               2 (1)
             21 INPLACE_ADD
             22 STORE_FAST               0 (x)

  5          25 LOAD_FAST                0 (x)
             28 YIELD_VALUE
             29 POP_TOP
             30 JUMP_ABSOLUTE            9
        >>   33 POP_BLOCK
        >>   34 LOAD_CONST               0 (None)
             37 RETURN_VALUE

To do it the way you have in mind, the horrors just start ;-) if you think about how to create an object to mimic just this:

def test():
    yield 2
    yield 3
    yield 4

Now your next() method would have to carry additional hidden state just to remember which yield comes next. Wrap that in some nested loops with some conditionals, and "unrolling" it into a single-entry next() becomes a nightmare.

This is exactly the answer I was looking for. I didn't know exactly how to tell what the language was doing under the hood. I'm glad you used the `dis` module - i knew what I was trying to look for, but couldn't come up with the phrasing to actually find the results. — Esteban, Sep 16 '16 at 03:24
In that case, I think you'll find it profitable to read the PEP in which generators were first proposed - you grasp enough now to make sense of most of it :-) https://www.python.org/dev/peps/pep-0255/ — Tim Peters, Sep 16 '16 at 03:42

score 1 · Answer 3 · edited May 23 '17 at 12:19

Are generators in python just a convenience wrapper for the user to make an iterator object?

No. A generator is a function, where as iterators are class. Hence, generator can not be a object of iterator. But in some way you can say that generator is a simplified approach to get iterator like capability. It means:

All Generators are iterators, but not all iterators are generators.

I will strongly suggest you refer below wiki links:

Iterator - traverses a collection one at a time
Generator - generates a sequence, one item at a time

An iterator is typically something that has a next method to get the next element from a stream. A generator is an iterator that is tied to a function.

I will suggest you to refer: Difference between Python's Generators and Iterators.

Do generators simply make new objects with __iter__ and next functions?

3 Answers3

Do generators simply make new objects with iter and next functions?