Asyncio not running Aiohttp requests in parallel

Question

I want to run many HTTP requests in parallel using python. I tried this module named aiohttp with asyncio.

import aiohttp
import asyncio

async def main():
    async with aiohttp.ClientSession() as session:
        for i in range(10):
            async with session.get('https://httpbin.org/get') as response:
                html = await response.text()
                print('done' + str(i))

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

I expect it to execute all the requests in parallel, but they are executed one by one. Although, I later solved this using threading, but I would like to know what's wrong with this?

Related : https://docs.aiohttp.org/en/stable/client_quickstart.html — Mooncrater, Nov 13 '21 at 16:12

Krishna Chaurasia · Answer 1 · 2021-03-20T15:54:04.537

16

You need to make the requests in a concurrent manner. Currently, you have a single task defined by main() and so the http requests are run in a serial manner for that task.

You could also consider using asyncio.run() if you are using Python version 3.7+ that abstracts out creation of event loop:

import aiohttp
import asyncio

async def getResponse(session, i):
    async with session.get('https://httpbin.org/get') as response:
        html = await response.text()
        print('done' + str(i))

async def main():
    async with aiohttp.ClientSession() as session:
        tasks = [getResponse(session, i) for i in range(10)] # create list of tasks
        await asyncio.gather(*tasks) # execute them in concurrent manner

asyncio.run(main())

edited Mar 20 '21 at 15:54

answered Mar 20 '21 at 08:21

Krishna Chaurasia

8,924
6
22
35

1

You can simplify the loop with list comprehension `tasks = [asyncio.create_task(getResponse(session, i)) for i in range(10)]` – Carlo Zanocco Mar 20 '21 at 08:24
1

thanks for the feedback, list comprehension should definitely be preferred. – Krishna Chaurasia Mar 20 '21 at 08:26
1

Understood! Thank you so much! – Anshit Bhardwaj Mar 20 '21 at 09:12
1

Unless you specifically need the `Task` associated with each `getResponse`, you can omit `create_task`. `[getResponse(session, i) for i in range(10)]` – dirn Mar 20 '21 at 13:30
I have used the above code.. once all the execution completed. again it executing and showing errors – Suganthan Raj Dec 15 '22 at 10:44

Asyncio not running Aiohttp requests in parallel

1 Answers1

Linked

Related