0

I need to download around 200 files from 200 urls in a very short time and very frequently. After countless attempts i came up with the code which is capable of downloading all files in a few seconds. Now i need to make sure that every file will get downloaded, no matter what. So in case of an error, i need to retry for one specific url (or all if it is not possible otherwise)

Common Errors are:

aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host ssl:None [None]
aiohttp.client_exceptions.ClientOSError: Cannot write to closing transport

  File "C:/test/test.py", line 14, in main
    loop.run_until_complete(downloader.download_data_async())
  File "C:\Users\Michael\AppData\Local\Programs\Python\Python37-32\lib\asyncio\base_events.py", line 584, in run_until_complete
    return future.result()
  File "C:\test\downloader.py", line 25, in download_data_async
    responses = await asyncio.gather(*tasks)
  File "C:\test\downloader.py", line 54, in fetch_async
    async with session.get(url) as response:
  File "C:\Users\Michael\AppData\Local\Programs\Python\Python37-32\lib\site-packages\aiohttp\client.py", line 1005, in __aenter__
    self._resp = await self._coro
  File "C:\Users\Michael\AppData\Local\Programs\Python\Python37-32\lib\site-packages\aiohttp\client.py", line 476, in _request
    timeout=real_timeout
  File "C:\Users\Michael\AppData\Local\Programs\Python\Python37-32\lib\site-packages\aiohttp\connector.py", line 522, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "C:\Users\Michael\AppData\Local\Programs\Python\Python37-32\lib\site-packages\aiohttp\connector.py", line 854, in _create_connection
    req, traces, timeout)
  File "C:\Users\Michael\AppData\Local\Programs\Python\Python37-32\lib\site-packages\aiohttp\connector.py", line 992, in _create_direct_connection
    raise last_exc
  File "C:\Users\Michael\AppData\Local\Programs\Python\Python37-32\lib\site-packages\aiohttp\connector.py", line 974, in _create_direct_connection
    req=req, client_error=client_error)
  File "C:\Users\Michael\AppData\Local\Programs\Python\Python37-32\lib\site-packages\aiohttp\connector.py", line 931, in _wrap_create_connection
    raise client_error(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host steamcommunity.com:443 ssl:None [None]

I have already tried surrounding the 'def fetch_async' with a try catch phrase, which did not work out.

from aiohttp import ClientSession
import asyncio

async def fetch_async(url, session):
    async with session.get(url) as response:
        return await response.text()


async def download_data_async():
    tasks = []
    #List of urls to download
    urls = [...]
    async with ClientSession() as session:
        for url in urls:
            task = asyncio.ensure_future(self.fetch_async(url,session))
            tasks.append(task)
        responses = await asyncio.gather(*tasks)
        return responses
Th3Fi3nD
  • 67
  • 6
  • Did you surround the *definition* of the `fetch_async` coroutine with a try catch? In that case try surrounding your call to the coroutine instead. Exception occur if you call something, not when you define it. Also a traceback would be helpful to pinpoint where the exception should be catched (and some self contained code, it looks like you removed a bit to much e.g. download_data_sync has a `self` argument, which seems strange without class). – syntonym Aug 13 '19 at 22:04
  • I tried surrounding everything inside the fetch_async definition. Traceback is added, Yes this is inside a "downloader" class, i changed it now. – Th3Fi3nD Aug 13 '19 at 22:14
  • From your traceback it seems to me that a try catch inside the fetch_async should work. I could get your code to run (with some minor changes, there is still a self in there and the asyncio boilerplate) but unfortunately couldn't reproduce your exception, because I don't know a website which would reliably close the http connection. – syntonym Aug 13 '19 at 22:49
  • Does this answer your question? https://stackoverflow.com/questions/63347818/aiohttp-client-exceptions-clientconnectorerror-cannot-connect-to-host-stackover – 0dminnimda Jan 09 '21 at 20:08

0 Answers0