1

I have an async generator that I'm trying to pass into a keras model fit_generator, but the async generator returns an object, not a generator.

I've tried googling, but I haven't found a solution. This seems to be a pretty specific problem.

starter1011
  • 35
  • 8
  • Have you tried collecting the generator into a list and passing that into the model? You can use an async list comprehension for that, e.g. `lst = [x async for x in bla()]` – user4815162342 Sep 08 '19 at 18:10
  • Yes, however, I have tens of millions of rows of data so I was trying to find a way to do async generator -> generator. – starter1011 Sep 08 '19 at 18:17
  • Can you do it in batches? Barring magic with threads, non-async code will never be able to handle an async generator. – user4815162342 Sep 08 '19 at 21:28
  • See [this answer](https://stackoverflow.com/a/55164899/1600898) for the threaded approach, but batching is probably a simpler option. – user4815162342 Sep 08 '19 at 21:35

1 Answers1

1

It's intention of asyncio to split async generator from regular generator, read answer here.

However if you decided that you won't run async generator concurrently elsewhere and your only intention is to avoid RAM overflow, you can cast async generator to regular one manually iterating async generator and awaiting each new item:

import asyncio


async def my_gen():
    for i in range(10):
        yield i
        await asyncio.sleep(0.5)


def to_sync_generator(ait):
    loop = asyncio.get_event_loop()
    try:
        while True:
            try:
                coro = ait.__anext__()
                res = loop.run_until_complete(coro)
            except StopAsyncIteration:
                return
            else:
                yield res
    finally:
        coro = loop.shutdown_asyncgens()
        loop.run_until_complete(coro)


# Check:

if __name__ == '__main__':
    for i in to_sync_generator(my_gen()):
        print(i)

P.S. Didn't test code much.

Mikhail Gerasimov
  • 36,989
  • 16
  • 116
  • 159