Modifying yield from's return value

Question

Let's say I have these parsers:

parsers = {
    ".foo": parse_foo,
    ".bar", parse_bar
}

parse_foo and parse_bar are both generators that yield rows one by one. If I wish to create a single dispatch function, I would do this:

def parse(ext):
    yield from parsers[ext]()

The yield from syntax allows me to tunnel information easily up and down the generators.

Is there any way to maintain the tunneling while modifying the yield results?
Doing so while breaking the tunneling is easy:

def parse(ext):
    for result in parsers[ext]():
        # Add the extension to the result
        result.ext = ext
        yield result

But this way I can't use .send() or .throw() all the way to the parser.

The only way I'm thinking of is by doing something ugly like try: ... except Exception: ... and pass the exceptions up, while doing the same for .send(). It's ugly, messy and bug-prone.

I think your best bet would probably be to implement a `passthrough_map` that does what `map` does while passing `send` and `throw` through to the generator you're mapping over. IIRC, doing that right is tricky, but you only need to get it right once, and then you can reuse it whenever you need that functionality. — user2357112, Apr 12 '16 at 20:37

Tamas Hegedus · Answer 1 · 2016-04-12T21:18:52.747

There is another way doing this besides try ... yield ... except: by implementing a new generator. With this class you can transform all the inputs and outputs of your underlying generator:

identity = lambda x: x
class map_generator:
  def __init__(self, generator, outfn = identity,
      infn = identity, throwfn = identity):
    self.generator = generator
    self.outfn = outfn
    self.infn = infn
    self.throwfn = throwfn
    self.first = True
  def __iter__(self):
    return self
  def __next__(self):
    return self.send(None)
  def _transform(self, value):
    if self.first:
      self.first = False
      return value
    else:
      return self.infn(value)
  def send(self, value):
    return self.outfn(self.generator.send(self._transform(value)))
  def throw(self, value):
    return self.outfn(self.generator.throw(self.throwfn(value)))
  def next(self): # for python2 support
    return self.__next__()

Usage:

def foo():
  for i in "123":
    print("sent to foo: ", (yield i))

def bar():
  dupe = lambda x:2*x
  tripe = lambda x:3*x
  yield from map_generator(foo(), dupe, tripe)

i = bar()
print("received from bar: ", i.send(None))
print("received from bar: ", i.send("B"))
print("received from bar: ", i.send("C"))

...

received from bar:  11
sent to foo:  BBB
received from bar:  22
sent to foo:  CCC
received from bar:  33

EDIT: You might want to inherit from collections.Iterator, but it is not neccessary in this usecase.

Thanks very much for the time you took answering me. I found a package called [cotoolz](https://pypi.python.org/pypi/cotoolz) that apparently does exactly that, but it is implemented in C thus resulting in faster execution, so I'll go with it. — Bharel, Apr 13 '16 at 18:12

Zach Gates · Answer 2 · 2016-04-12T20:28:02.553

0

Have parse_foo and parse_bar add the extensions:

def parse_foo(ext):
    # Existing code
    ...
    # Add an extension to the item(s)
    item.ext = ext

def parse(ext):
    yield from parsers[ext](ext)

Or just hardcode it in each function:

def parse_foo():
    # Existing code
    ...
    # Add an extension to the item(s)
    item.ext = ".foo"

edited Apr 12 '16 at 20:28

answered Apr 12 '16 at 20:04

Zach Gates

4,045
1
27
51

This breaks `send` and `throw`. – user2357112 Apr 12 '16 at 20:08
This does not work and indeed breaks `send` and `throw`. – Bharel Apr 12 '16 at 20:09
@user2357112 In what way does this break `send` and `throw`? – Zach Gates Apr 12 '16 at 20:15
@ZachGates: With `yield from`, a `send` or `throw` on the outer generator gets passed through to the generator you're `yield from`-ing. Putting a `map` in the way stops that from working. – user2357112 Apr 12 '16 at 20:20
@user2357112: Thanks for the explanation; I believe I've fixed it. – Zach Gates Apr 12 '16 at 20:31

Bharel · Accepted Answer · 2016-04-13T18:23:49.777

Unfortunately there is no built-in that does it. You may implement it yourself using classes but a package called cotoolz implements a map() function that does exactly that.

Their map function is 4 times slower than the builtin map() but it's aware to the generator protocol, and faster than a similar Python implementation (it's written in C and requires a C99 compiler).

An example from their page:

>>> def my_coroutine():
...     yield (yield (yield 1))
>>> from cotoolz import comap
>>> cm = comap(lambda a: a + 1, my_coroutine())
>>> next(cm)
2
>>> cm.send(2)
3
>>> cm.send(3)
4
>>> cm.send(4)
Traceback (most recent call last):
    ...
StopIteration

Modifying yield from's return value

3 Answers3