Asynchronous HTTP calls using aiohttp/asyncio fail with "Cannot connect to host [Network is unreachable]"

Question

I am implementing a fast async REST interface caller with which I want to upload data in an synchronous manner. For now I am about to build the async framework that just calls the python page in an async manner and measures the delay.

Here is the code (which does not work as described below):

import aiohttp
import asyncio
import async_timeout
from timeit import default_timer as timer

async def fetch(session, url):
    start = timer()
    with async_timeout.timeout(10):
        async with session.get(url) as response:
            date = response.headers.get("DATE")
            end = timer()
            delay = end - start
            print("{}:{} with delay {} s".format(date, response.url, delay))
            return await response.read()

async def bound_call(semaphore, session, url):
    async with semaphore:
        await fetch(session, url)

async def perform_call(session):
    sem = asyncio.Semaphore(1000)
    html = await bound_call(sem, session, 'http://python.org')

async def perform_calls(n):
    tasks = []
    async with aiohttp.ClientSession() as session:
        for i in range(n):
            task = perform_call(session)
            tasks.append(task)

        responses = asyncio.gather(*tasks)
        await responses

call_number = 10

loop = asyncio.get_event_loop()
loop.run_until_complete(perform_calls(call_number))

It properly works as long as I am using

await perform_call(session)

but that obviously breaks asynchonous calls. If I replace it with

            task = perform_call(session)
            tasks.append(task)

        responses = asyncio.gather(*tasks)
        await responses

to make it await all responses at the same time, I get the error below:

aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host python.org:80 ssl:False [Network is unreachable]

I tried to run this as code from a jupyter notebook, and, since it did not run, copied it into normal code. In both cases, it ran on python 3.5. Unfortunately, I could not find a solution to the problem. It seems that there is no network access as soon as I try to use gather. Does anybody have a suggestion why it does not work? I am happy for any suggestions.

Answer 1

Ok, I found the answer in a question my question is a duplicate of (I only found the duplicate after I posted here.)

The solution is just to initialize the client session using:

import socket # together with your other imports

conn = aiohttp.TCPConnector(
        family=socket.AF_INET,
        verify_ssl=False,
    )

   # Create client session that will ensure we dont open new connection
   # per each request.
   async with aiohttp.ClientSession(connector=conn) as session:

Explanation based on the duplicate: The client session will use the connection instead of the default AsyncResolver as the resolver for the connection. It used to be the default resolver. The problem seems to be related to domains with ipv6 where the AsyncResolver has problems, so the solution is to simply specify the family to ipv4 addresses which is what we do with family=socket.AF_INET.

Duplicate of: python 3.5 asyncio and aiohttp Errno 101 Network is unreachable