30

So I need to call an async function for all items in a list. This could be a list of URLs and an async function using aiohttp that gets a response back from every URL. Now obviously I cannot do the following:

async for url in ['www.google.com', 'www.youtube.com', 'www.aol.com']:

I can use a normal for loop but then my code will act synchronously and I lose the benefits and speed of having an async response fetching function.

Is there any way I can convert a list such that the above works? I just need to change the list's __iter__() to a __aiter__() method right? Can this be achieved by subclassing a list? Maybe encapsulating it in a class?

sophros
  • 14,672
  • 11
  • 46
  • 75
Max Smith
  • 925
  • 1
  • 14
  • 25

2 Answers2

34

Use asyncio.as_completed:

for future in asyncio.as_completed(map(fetch, urls)):
    result = await future

Or asyncio.gather:

results = await asyncio.gather(*map(fetch, urls))

EDIT: If you don't mind having an external dependency, you can use aiostream.stream.map:

from aiostream import stream, pipe

async def fetch_many(urls):
    xs = stream.iterate(urls) | pipe.map(fetch, ordered=True, task_limit=10)
    async for result in xs:
        print(result)

You can control the amount of fetch coroutine running concurrently using the task_limit argument, and choose whether to get the results in order, or as soon as possible.

See more examples in this demonstration and the documentation.

Disclaimer: I am the project maintainer.

Vincent
  • 12,919
  • 1
  • 42
  • 64
  • 5
    But here you are using map and that is essentially like a normal for loop, applying the function to every element in the list. Why would this be any faster and how is it async? Map is also limited by the number of parameters the fetch function can receive. I was planning on passing an aiohttp session as well. – Max Smith Jan 01 '18 at 19:08
  • 1
    @MaxSmith In this case, `map` simply creates the coroutines. Then `as_completed` (or `gather`, or `wait`) schedules them so they can run concurrently. An `async for` loop is not needed, since the asynchronous call is done within the loop (and not in the the `iter`/`next` calls) – Vincent Jan 01 '18 at 19:11
  • Thank you, this seems to work well except that now the program crashes because there are too many open sockets... I think I'm going to need semaphores... – Max Smith Jan 01 '18 at 20:42
  • 2
    @MaxSmith See my edit for a solution to your socket issue. – Vincent Jan 05 '18 at 15:58
  • @Vincent can you please check here? https://stackoverflow.com/questions/73855591/python-async-function-proper-syntax?noredirect=1#comment130411246_73855591 – automation_m Sep 26 '22 at 18:57
13

Please note, that Vincents answer has a partial problem:
You must have a splatter operator infront of the map function, otherwise asyncio.gather would try to use the list as whole. So do it like this:

results = await asyncio.gather(*map(fetch, url))
quadronom
  • 652
  • 1
  • 5
  • 15