Combining multiple infinite generator objects into one generator

Question

The aim is to read the input from multiple Gamepads and handle it within Python. I am using the evdev library.

Assuming a given device object dev the events the Gamepad sends can easily be read by using the function dev.read_loop() according to the documentation. This function call returns a generator object told by my debugger. Then a simple for loop can be used to iterate the generator. It will block until a new event occurs and then the event is processed with whatever code one uses within the for loop.

Now I want to bundle this event "stream" from multiple Gamepads using a custom class (which does some more like assigning them ID's, keeping track of disconnects, etc.) and this is where my problem comes up. I cannot find a way to create one generator like this, that combines the events from all Gamepads. The Gamepad objects are given in a list.

The provided example code tries to mimic the desired behaviour but fails. The constructor prints all available devices, then initializes them. The method global_generator() is my approach to this problem. The issue is, that the outer for loop for device in self.__gamepads never proceeds as the first generator is endless and thus will always and only forward the events from the first controller. I tried to use itertools.chain() but they have the same problem as the first generator never ends. So what I want is basically: "As soon as one of these n generators yields a result, yield it further (and thus combine them)". All ideas I have, like creating queues or anything else always cease due to the problem that I cannot simultaneously loop my devices, thus their generators.

I've seen the part in the documentation where it shows how to read from multiple devices. The problem is that they always print the result but do not show how to pass it on in a way I want.

import hardware.ControllerHandler as ch

controller_handler = ch.ControllerHandler()
global_generator = controller_handler.global_generator()

for event in global_generator:
    #event = ce.ControllerEvent(event)
    print(event)

import evdev
from evdev import InputDevice
import asyncio


class ControllerHandler:

    def __init__(self):
        print(evdev.list_devices())
        self.__gamepads = list(map(InputDevice, evdev.list_devices()))

    @property
    def gamepads(self):
        return self.__gamepads

    @property
    def count(self):
        return len(self.__gamepads)

    async def global_generator(self):
        for device in self.__gamepads:
            async for event in device.async_read_loop():
                yield event

So the expected result is a generator that returns any of the underlying events, not just the events from the first Gamepad.

Not sure how much this helps, but are you looking for something like [`asyncio.wait`](https://docs.python.org/3/library/asyncio-task.html#asyncio.wait) with `FIRST_COMPLETED`? — Norrius, Jul 19 '19 at 23:58
@Norrius The problem is that the OP wants to combine multiple async iterables (in this case generators), not just poll individual futures. This requires obtaining the async iterators using `__aiter__()` method and calling `wait(FIRST_COMPLETED)` on the futures returned by their respective `__anext__()`, then transmitting the result(s) and handling `StopAsyncIteration`. [This answer](https://stackoverflow.com/a/50903757/1600898) shows an implementation of that approach. — user4815162342, Jul 20 '19 at 08:04

user4815162342 · Accepted Answer · 2019-07-20T09:35:09.757

3

You can use the aiostream library to merge multiple async generators:

    async def global_generator(self):
        loops = [device.async_read_loop() for device in self.__gamepads]
        async with aiostream.stream.merge(*loops).stream() as merged:
            async for event in merged:
                yield event

If you don't like to depend on aiostream, see this answer for an explanation how to merge streams using pure asyncio.

Note that, since the resulting generator is asynchronous, you will need to iterate over it using async for inside a coroutine.

edited Jul 20 '19 at 09:35

answered Jul 20 '19 at 07:22

user4815162342

141,790
18
296
355

1

Thank your very much user4815162342, this was exactly what I needed! One more question: This returns an async generator, is it possible to also return a normal (thus not async) generator, if yes, how? Also thanks for the link, didn't expect it to be that "difficult" without using an external library. As it is just a hobby project, I am fine with using `aiostream`. – Muqq Jul 20 '19 at 11:35
1

@Muqq Normally it has to be async. You could avoid that, but you'd have to run the event loop in a separate thread. – user4815162342 Jul 20 '19 at 14:29
I could also obtain synchronous generators using `read_loop()`. Is there a way to combine multiple synchronous, but infinite, generators to one synchronous one? – Muqq Jul 20 '19 at 14:52
1

@Muqq Only at the cost of creating as many threads as there are generatora. The idea behind asyncio is that it allows parallelizing code without having to create threads for purposes like this. If the library you are using supports asyncio, why not make use of it? – user4815162342 Jul 20 '19 at 15:58

Combining multiple infinite generator objects into one generator

1 Answers1