get completed aiohttp parallel requests after timeout

Question

I'm using aiohttp to perform some parallel HTTP post requests.

I have to set a timeout globally (on ClientSession) in order to not exceed a threshold.

The problem is that I would like to take the (partial in the sessions) responses that I have completed before the threshold, so for example if the session contains 10 requests and before the timeout I've completed 5 of these, I want to take the result of these 5. But I've not figured out how to do this.

The code I'm using is something like that:

import aiohttp
import asyncio
import requests

async def fetch(session):
    async with session.get("https://amazon.com") as response:
        return response.status

async def main(n, timeout):
    async with aiohttp.ClientSession(timeout=timeout) as session:
        return await asyncio.gather(*(fetch(session) for _ in range(n)))

timeout = aiohttp.ClientTimeout(total=0.4)
res = asyncio.run(main(10, timeout))
print(res)

With timeout = 0.4 it raises asyncio.TimeoutError and I don't know how to get the partial performed responses.

For example, if I set the timeout at 5 seconds, all requests are completed and I obtain a list of ten 200.

Thank you

Pynchia · Accepted Answer · 2020-01-23T11:14:50.220

1

Use asyncio.wait instead of asyncio.gather

Also see this QA for further info on the differences.

Note: wait's timeout argument is expressed in seconds.

Most of all, you might not need to specify a timeout to the ClientSession at all.

The reworked code (for increased variance in the response time I have added a few different sources and 20 requests are performed)

import asyncio
import random
import aiohttp
import requests

sources = ["amazon.com", "hotmail.com", "stackoverflow.com"]

async def fetch(session):
    rnd = random.choice(sources)
    async with session.get(f"https://{rnd}") as response:
        return response.status

async def main(n, timeout):
    async with aiohttp.ClientSession() as session:
        completed, pending = await asyncio.wait(
            [fetch(session) for _ in range(n)],
            timeout=timeout
        )
    for t in pending:  # cancel the pending tasks
        t.cancel()
    return [t.result() for t in completed]

timeout = 0.5
res = asyncio.run(main(20, timeout))
print(res)

with increasing values of timeout as 0.3, 0.5 and 0.8 produces

(.venv) async_req_timeout $ python async_req_timeout.py 
[200, 200]

(.venv) async_req_timeout $ python async_req_timeout.py 
[200, 200, 200, 200, 200, 200, 200, 200, 200, 200]

(.venv) (base) async_req_timeout $ python async_req_timeout.py 
[200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200]

edited Jan 23 '20 at 11:14

answered Jan 23 '20 at 09:35

Pynchia

10,996
5
34
43

With this I have `TypeError: wait() takes 1 positional argument but 10 positional arguments (and 1 keyword-only argument) were given`, I think is in `*(fetch(session) for _ in range(n))` – rivamarco Jan 23 '20 at 09:51
Yes of course, no need to unpack the arguments: pass it the generator expression only. Corrected – Pynchia Jan 23 '20 at 09:56
It doesn't work. With `(fetch(session) for _ in range(n))` there is an error that i can correct using `[fetch(session) for _ in range(n)]`. If I put timeout in `asyncio.wait` I obtain `Session is closed`, putting it in `ClientSession` works but if it goes in timeout I'm not able to get the 200 responses. If it completes before the timeout I obtain the list of 200 – rivamarco Jan 23 '20 at 10:12
1

The two timeouts differ in representation: one is a ClientTimeout class, one is an int/float, expressed in seconds, see my note above. Try creating ClientSession without the timeout and let timeout be an int or a float instead of ClientTimeout. That is to say, use the timeout on `wait` only. Or use a longer timeout on the ClientSession and a shorter one on `wait` – Pynchia Jan 23 '20 at 10:27
With `[fetch(session) for _ in range(n)]` instead of `(fetch(session) for _ in range(n))` works! I can't change edit it because it's too little. After that, I will approve. Thank you – rivamarco Jan 23 '20 at 10:42
Good. I have tried myself, I can edit my answer and post my results – Pynchia Jan 23 '20 at 10:48
1

Please see the final answer, which now includes the full code – Pynchia Jan 23 '20 at 11:00

get completed aiohttp parallel requests after timeout

1 Answers1