Attempting to understand yield as an expression

Question

I'm playing around with generators and generator expressions and I'm not completely sure that I understand how they work (some reference material):

>>> a = (x for x in range(10))
>>> next(a)
0
>>> next(a)
1
>>> a.send(-1)
2
>>> next(a)
3

So it looks like generator.send was ignored. That makes sense (I guess) because there is no explicit yield expression to catch the sent information ...

However,

>>> a = ((yield x) for x in range(10))
>>> next(a)
0
>>> print next(a)
None
>>> print next(a)
1
>>> print next(a)
None
>>> a.send(-1)  #this send is ignored, Why? ... there's a yield to catch it...
2
>>> print next(a)
None
>>> print next(a)
3
>>> a.send(-1)  #this send isn't ignored
-1

I understand this is pretty far out there, and I (currently) can't think of a use-case for this (so don't ask;)

I'm mostly just exploring to try to figure out how these various generator methods work (and how generator expressions work in general). Why does my second example alternate between yielding a sensible value and None? Also, Can anyone explain why one of my generator.send's was ignored while the other wasn't?

Answer 1

The confusion here is that the generator expression is doing a hidden yield. Here it is in function form:

def foo():
    for x in range(10):
        yield (yield x)

When you do a .send(), what happens is the inner yield x gets executed, which yields x. Then the expression evaluates to the value of the .send, and the next yield yields that. Here it is in clearer form:

def foo():
    for x in range(10):
        sent_value = (yield x)
        yield sent_value

Thus the output is very predictable:

>>> a = foo()
#start it off
>>> a.next() 
0
#execution has now paused at "sent_value = ?"
#now we fill in the "?". whatever we send here will be immediately yielded.
>>> a.send("yieldnow") 
'yieldnow'
#execution is now paused at the 'yield sent_value' expression
#as this is not assigned to anything, whatever is sent now will be lost
>>> a.send("this is lost") 
1
#now we're back where we were at the 'yieldnow' point of the code
>>> a.send("yieldnow") 
'yieldnow'
#etc, the loop continues
>>> a.send("this is lost")
2
>>> a.send("yieldnow")
'yieldnow'
>>> a.send("this is lost")
3
>>> a.send("yieldnow")
'yieldnow'

EDIT: Example usage. By far the coolest one I've seen so far is twisted's inlineCallbacks function. See here for an article explaining it. The nub of it is it lets you yield functions to be run in threads, and once the functions are done, twisted sends the result of the function back into your code. Thus you can write code that heavily relies on threads in a very linear and intuitive manner, instead of having to write tons of little functions all over the place.

See the PEP 342 for more info on the rationale of having .send work with potential use cases (the twisted example I provided is an example of the boon to asynchronous I/O this change offered).

Answer 2

You're confusing yourself a bit because you actually are generating from two sources: the generator expression (... for x in range(10)) is one generator, but you create another source with the yield. You can see that if do list(a) you'll get [0, None, 1, None, 2, None, 3, None, 4, None, 5, None, 6, None, 7, None, 8, None, 9, None].

Your code is equivalent to this:

>>> def gen():
...     for x in range(10):
...         yield (yield x)

Only the inner yield ("yield x") is "used" in the generator --- it is used as the value of the outer yield. So this generator iterates back and forth between yielding values of the range, and yielding whatever is "sent" to those yields. If you send something to the inner yield, you get it back, but if you happen to send on an even-numbered iteration, the send is sent to the outer yield and is ignored.

Answer 3

This generator translates into:

for i in xrange(10):
    x = (yield i)
    yield x

Result of second call to send()/next() are ignored, because you do nothing with result of one of yields.

Answer 4

The generator you wrote is equivalent to the more verbose:

def testing():
    for x in range(10):
            x = (yield x)
            yield x

As you can see here, the second yield, which is implicit in the generator expression, does not save the value you pass it, therefore depending on where the generator execution is blocked the send may or may not work.

Answer 5

Indeed - the send method is meant to work with a generator object that is the result of a co-routine you have explicitly written. It is difficult to get some meaning to it in a generator expression - though it works.

-- EDIT -- I had previously written this, but it is incorrecct, as yield inside generator expressions are predictable across implementations - though not mentioned in any PEP.

generator expressions are not meant to have the yield keyword - I am not shure the behavior is even defined in this case. We could think a little and get to what is happening on your expression, to meet from where those "None"s are coming from. However, assume that as a side effect of how the yield is implemented in Python (and probably it is even implementation dependent), not as something that should be so.

The correct form for a generator expression, in a simplified manner is:

(<expr> for <variable> in <sequence> [if <expr>])

so, <expr> is evaluated for each value in the <sequence: - not only is yield uneeded, as you should not use it.

Both yield and the send methods are meant to be used in full co-routines, something like:

def doubler():
   value = 0
   while value < 100:
       value = 2 * (yield value)

And you can use it like:

>>> a = doubler()
>>> # Next have to be called once, so the code will run up to the first "yield"
... 
>>> a.next()
0
>>> a.send(10)
20
>>> a.send(20)
40
>>> a.send(23)
46
>>> a.send(51)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration
>>>