8

Below is a generator function.

def f():
   x=1
   while 1:
      y = yield x
      x += y

Does this generator function (f) get implemented, internally, as shown below?

class f(collections.Iterable):
   def __init__(self):
      self.x = 1
   def __iter__(self):
      return iter(self)
   def __next__(self):
      return self.x
   def send(self, y):
      self.x += y
      return self.next()

Edit:

This is the answer for my question.

Raymond Hettinger
  • 216,523
  • 63
  • 388
  • 485
overexchange
  • 15,768
  • 30
  • 152
  • 347
  • 1
    You can test whether they *behave* the same for yourself. Going into the internal implementation details seems too broad for a SO question. – jonrsharpe Aug 16 '17 at 22:25
  • 1
    This post may provide some info: http://aosabook.org/en/500L/a-web-crawler-with-asyncio-coroutines.html – Christian Dean Aug 16 '17 at 22:27
  • I doubt that those are equivelent... (they may behave the same) ... check out the `dis` module to see the bytecode implementation (as an aside as an implementation detail, the answer may change depending on the python interpretter being used) – Joran Beasley Aug 16 '17 at 22:28
  • 2
    of course they aren't equivalent, one is a class and one is a function – wim Aug 16 '17 at 22:29
  • @wim I pretty sure he meant are `f` and `f()` equivalent. Regardless, the answer is still no. – Christian Dean Aug 16 '17 at 22:31
  • 1
    Possible duplicate of [What does the "yield" keyword do?](https://stackoverflow.com/questions/231767/what-does-the-yield-keyword-do) – pppery Aug 17 '17 at 00:25
  • 2
    The short answer is that **generators are not implemented internally as shown in your pure python class**. Instead, they share most of the same logic as regular functions. – Raymond Hettinger Aug 17 '17 at 06:36
  • @RaymondHettinger On server side, does `await loop.sock_accept(listenSock)` use `yield` & `next()` internally, to wait for new data connection from client? – overexchange Aug 21 '17 at 22:42
  • @ppperry Commented in your referred question. – overexchange Aug 21 '17 at 22:57

1 Answers1

28

Internally, a generator works about the same as a regular function call. Under-the-hood, running generators and running functions use mostly the same machinery.

When you call either a function or a generator, a stackframe is created. It has the local variables (including the arguments passed into the function), a code pointer to the active opcode, and a stack for pending try-blocks, with-blocks, or loops.

In a regular function, execution begins immediately. When return is encountered, the final result is kept and the stackframe is freed along with everything it referenced.

In a generator function, the stackframe is wrapped in a generator-iterator object and returned immediately. The code in the generator function only runs when called by next(g) or g.send(v). Execution is suspended when yield is encountered.

One way to think of generators is that they are like functions that can be paused with yield and resumed with g.next(). The stackframe is kept alive, so resuming a running generator is much cheaper than making a new function call which has to build a new frame on every invocation.

Raymond Hettinger
  • 216,523
  • 63
  • 388
  • 485
  • 1
    How `yield` pause/suspend the function internally? – overexchange Aug 17 '17 at 06:14
  • 3
    @overexchange Execution of both generators and functions involves running the current opcode and updating the code pointer. Pausing just means to stop doing that. Resuming means to continue doing that. Count with me, 1, 2, 3, and now talk about something else, and continue couting 4, 5, 6, ... All you need to know is the last count. Likewise, the stackframe keeps the state of the function and you can resume stop updating it and resume updating it at any time. – Raymond Hettinger Aug 17 '17 at 06:20
  • Building block for async is about task capable to suspend/resume. For server side programming, generally task is suspended on performing IO. When u say, pausing is to stop, are we not pausing because we are waiting for an IO(say http get request)? – overexchange Aug 17 '17 at 06:25
  • 8
    @overexchange I think you're imagining this to be more complex than it is. The code for ``next(g)`` boils down to basically ``PyEval_EvalFrameEx(gen->gi_frame, exc)``. That's it, it just execs the current state of the frame (see ``Objects/genobject.c`` which is clear and simple in Python 2.7). A ``yield`` simply returns from that call. In contrast, a function call runs ``PyEval_EvalCodeEx(...)`` which creates a new frame and execs it as shown above. See ``Objects/funcobject.c``. – Raymond Hettinger Aug 17 '17 at 06:34