Create a generator that yields values from any number of inner generators

Question

I have a generator function generate which yields 5 random numbers one at a time. I need to be able to generate the numbers in two ways:

Single generation, which means a single output of generate function
Multiple generation, which means multiple execution of generate and yielding all the results together as a single (merged) flow

For that I wrote another function get_resource, which calls generate either once or using itertools.chain to run the generators one after another, but transparently to the caller.

My goal is to use get_resource function and produce the results in the same format (one list of numbers), regardless of single/multiple generations.

import itertools
import random


def get_resource(values=None):
    def resource_generator():
        if values:
            # run a generator for every value
            return itertools.chain(generate(value) for value in values)
        else:
            return generate('meh')

    return resource_generator()


def generate(value):
    for _ in range(5):
        yield random.randint(1, 101)


if __name__ == '__main__':
    # list() is used for convenience only, 
    # I still need the values one by one
    print list(get_resource())
    print list(get_resource([1, 2, 3]))

It prints:

[63, 22, 87, 2, 54]
[<generator object generate at 0x1089f7640>, <generator object generate at 0x1089f7690>, <generator object generate at 0x1089f76e0>]

While I need it to print:

[63, 22, 87, 2, 54]
[63, 22, 87, 2, 54, 1, 58, 79, 89, 77, 94, 99, 30, 30, 4]

I use python2.7

Why do you define `resource_generator()` instead of just putting its code directly inside `get_resource()` ? — khelwood, Feb 19 '18 at 09:07
@khelwood, it is a simplified and minimal version of my code, which has some more things going on around. In this example it is redundant, while I still need it in my code — Yury Fedorov, Feb 19 '18 at 09:08
For 2.7, you will need a nested loop with yield, I've edited my answer accordingly. Although I'd recommend just upgrading, if you can. — cs95, Feb 19 '18 at 09:17

cs95 · Answer 1 · 2018-02-19T16:37:34.727

5

You can specify generator delegation using yield from starting with python-3.3+.

def get_resource(values=None):
    def resource_generator():
        if values:
            for value in values:
                yield from generate(value)
        else:
            yield from generate(None)

    return resource_generator()

Now,

>>> list(get_resource([1, 2, 3]))
[46, 99, 97, 1, 42, 79, 69, 9, 45, 25, 77, 56, 54, 7, 41]

edited Feb 19 '18 at 16:37

answered Feb 19 '18 at 09:06

cs95

379,657
97
704
746

I tried something similar to your python2 code, but this syntax is not supported before python3.3: `SyntaxError: 'return' with argument inside generator` – Yury Fedorov Feb 19 '18 at 09:24
@YuryFedorov Hmm, my bad, I didn't test it on python3.7, and assumed it would work as is. I've removed that part of my answer. :) – cs95 Feb 19 '18 at 09:26
shouldn't the `return generate(None)` be `yield from generate(None)`? a return in a generator triggers a `StopIteration` – Maarten Fabré Feb 19 '18 at 11:36
@MaartenFabré this is okay with 3.6. I'm not sure when it starts to break. I would need to test some more. – cs95 Feb 19 '18 at 16:13
https://docs.python.org/3/reference/simple_stmts.html#the-return-statement `In a generator function, the return statement indicates that the generator is done and will cause StopIteration to be raised. The returned value (if any) is used as an argument to construct StopIteration and becomes the StopIteration.value attribute.` – Maarten Fabré Feb 19 '18 at 16:31
@MaartenFabré Keep in mind that the yield and return statements will not be executed in the same code path. – cs95 Feb 19 '18 at 16:32
I'm just trying to understand this. Doesn't the presence of the `yield` automatically makes this function a generator, function path or not? `list(get_resource())` returns `[]`, which seems to support the idea that the `return` triggers a `StopIteration` – Maarten Fabré Feb 19 '18 at 16:35
@MaartenFabré wow, what version are you on? Let me just fix it then. – cs95 Feb 19 '18 at 16:37
python 3.6; I'm not talking about this issue `If a generator function defined in the presence of a from __future__ import generator_stop directive raises StopIteration, it will be converted into a RuntimeError (retaining the StopIteration as the new exception’s cause).`, I'm just saying that `return` in a generator is syntactic sugar for `raise StopIteraton `, as also answere [here](https://stackoverflow.com/a/42169840/1562285). What does `list(get_resource())` return for you? – Maarten Fabré Feb 19 '18 at 16:50
@MaartenFabré You were right! My mistake for not testing my answer. I've fixed it to be `yield from` instead of return. Thanks for pointing it out :) – cs95 Feb 19 '18 at 22:25

score 2 · Accepted Answer · answered Feb 19 '18 at 09:08

2

you should use itertools.chain.from_iterable

return itertools.chain.from_iterable(generate(value) for value in values)

answered Feb 19 '18 at 09:08

Maarten Fabré

6,938
1
17
36

Create a generator that yields values from any number of inner generators

2 Answers2