aiohttp asyncio.TimeoutError from None using ClientSession

Question

It's a weird error since when I try/catch it, it prints nothings.

I'm using sanic server to asyncio.gather a bunch of images concurrently, more than 3 thousand images.

I haven't got this error when dealing with a smaller sample size.

Simplified example :

from sanic import Sanic
from sanic import response
from aiohttp import ClientSession

from asyncio import gather

app = Sanic()

@app.listener('before_server_start')
async def init(app, loop):
    app.session = ClientSession(loop=loop)

@app.route('/test')
async def test(request):
    data_tasks = []
    #The error only happened when a large amount of images were used
    for imageURL in request.json['images']:
        data_tasks.append(getRaw(imageURL))
    await gather(*data_tasks)
    return response.text('done')

async def getRaw(url):
    async with app.session.get(url) as resp:
        return await resp.read()

What could this error be? If it is some kind of limitation of my host/internet, how can I avoid it?

I'm using a basic droplet from DigitalOcean with 1vCPU and 1GB RAM if that helps

Full stack error :

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/sanic/app.py", line 750, in handle_request
    response = await response
  File "server-sanic.py", line 53, in xlsx
    await gather(*data_tasks)
  File "/usr/lib/python3.5/asyncio/futures.py", line 361, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/lib/python3.5/asyncio/tasks.py", line 296, in _wakeup
    future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/lib/python3.5/asyncio/tasks.py", line 241, in _step
    result = coro.throw(exc)
  File "server-sanic.py", line 102, in add_data_to_sheet
    await add_img_to_sheet(sheet, rowIndex, colIndex, val)
  File "server-sanic.py", line 114, in add_img_to_sheet
    image_data = BytesIO(await getRaw(imgUrl))
  File "server-sanic.py", line 138, in getRaw
    async with app.session.get(url) as resp:
  File "/usr/local/lib/python3.5/dist-packages/aiohttp/client.py", line 690, in __aenter__
    self._resp = yield from self._coro
  File "/usr/local/lib/python3.5/dist-packages/aiohttp/client.py", line 277, in _request
    yield from resp.start(conn, read_until_eof)
  File "/usr/local/lib/python3.5/dist-packages/aiohttp/client_reqrep.py", line 637, in start
    self._continue = None
  File "/usr/local/lib/python3.5/dist-packages/aiohttp/helpers.py", line 732, in __exit__
    raise asyncio.TimeoutError from None
concurrent.futures._base.TimeoutError

@yorodm I guess that's kinda hard to limit through python, should I just limit the max concurrent connections? — Mojimi, Feb 05 '19 at 20:08
Can you check the answers [here](https://stackoverflow.com/questions/48483348/limited-concurrency-with-asyncio) and see if those works for you? — yorodm, Feb 05 '19 at 20:12
@yorodm I think [this](https://docs.aiohttp.org/en/stable/client_advanced.html#limiting-connection-pool-size) might be enough? Limiting pool size in aiohttp — Mojimi, Feb 05 '19 at 20:14

score 7 · Accepted Answer · answered Feb 10 '19 at 20:00

7

There is no benefit to launching a million requests at once. Limit it to 10 or whatever works and wait for those before continuing the loop.

for imageURL in request.json['images']:
    data_tasks.append(getRaw(imageURL))
    if len(data_tasks) > 10:
        await gather(*data_tasks)
        data_tasks = []        
await gather(*data_tasks)

answered Feb 10 '19 at 20:00

Craftables

236
1
8

That does makes sense! Do you reckon this option from aiohttp is enough? https://docs.aiohttp.org/en/stable/client_advanced.html#limiting-connection-pool-size – Mojimi Feb 11 '19 at 11:03
@Craftables There surely is some benefit to issuing thousands of requests simultaneously if they are on distinct hosts ... – K3---rnc Jul 06 '23 at 11:14

aiohttp asyncio.TimeoutError from None using ClientSession

1 Answers1