13

I need to process several pages of data from server. I would like to make a generator for it like this. Unfortunately I get TypeError: 'async_generator' object is not iterable

async def get_data():
    i = 0
    while i < 3:
        i += 1
        data = await http_call()  # call to http-server here
        yield data

data = [i for i in get_data()]  # inside a loop

Next variant raises TypeError: object async_generator can't be used in 'await' expression

data = [i for i in await get_data()]  # inside a loop
SS_Rebelious
  • 1,274
  • 2
  • 19
  • 35

1 Answers1

24

Use async for in your comprehension. See PEP 530 -- Asynchronous Comprehensions

data = [i async for i in get_data()]

Depending on what version of Python you're using, this may only be available in async def functions.

Patrick Haugh
  • 59,226
  • 13
  • 88
  • 96
  • Note that this won't do async calls to all pages. – Aaron_ab Sep 20 '18 at 18:10
  • @Aaron_ab What do you mean? – Patrick Haugh Sep 20 '18 at 18:48
  • if we'll log every access to get_data, at start and end, we'll see that get_data1 will start and end, then get_data2 and so on.. no concurrency there – Aaron_ab Oct 21 '18 at 19:30
  • `async` doesn't mean "concurrent" as in preemptive multithreading. There *is* no preemptive multithreading at the pure-Python level, thanks to the GIL. `async` means [cooperative multitasking](https://en.wikipedia.org/wiki/Cooperative_multitasking), meaning the developer explicitly relinquishes control to other currently paused coroutines and asynchronous generators at each `await` expression. In the absence of explicit multithreading or multiprocessing, asynchronous generators still yield their results sequentially (i.e., non-concurrently). – Cecil Curry Sep 23 '21 at 02:32
  • If you want genuine concurrency under the asynchronous paradigm, consider either [the `asyncio.Queue` class](https://stackoverflow.com/a/52615705/2809027) and [`asyncio.as_completed()` function](https://stackoverflow.com/a/46789472/2809027) for I/O-bound operations *or* [the `async.run_in_executor()` function](https://stackoverflow.com/questions/52582685/using-asyncio-queue-for-producer-consumer-flow#comment95646656_52615705) for CPU-bound operations instead. – Cecil Curry Sep 23 '21 at 02:36