11

I'm using Python Asyncio to do a lot of HTTP requests.

I'm wondering what is the default concurrency level of Asyncio or how many HTTP requests would be happening in parallel at any given time?

The code I'm using to do the HTTP requests you can find below:

async def call_url(self, session, url):
    response = await session.request(method='GET', url=url)
    return response


async def main(self, url_list):
    async with aiohttp.ClientSession() as session:
        res = await asyncio.gather(*[self.call_url(session, url) for url in url_list])
        return res
fshabashev
  • 619
  • 6
  • 20
  • 1
    There is no default, available resources are the limiting factor. It's up to the developer to implement a mechanism such as a thread pool. Check out the asyncio-pool library: https://pypi.org/project/asyncio-pool/ – Dan Nagle Feb 20 '21 at 03:09
  • Does this answer your question? [How to limit concurrency with Python asyncio?](https://stackoverflow.com/questions/48483348/how-to-limit-concurrency-with-python-asyncio) – Sers Feb 20 '21 at 08:03
  • 1
    @Sers The question you linked is different. This question explicitly asks about the default limit on code that doesn't do its own limiting, which is not at all covered by the answers to the other one. – user4815162342 Feb 20 '21 at 11:18
  • 1
    @DanNagle aiohttp does limit to 100 parallel connections by default. – user4815162342 Feb 20 '21 at 11:22

1 Answers1

4

There is no built-in limit in asyncio, but there is one in aiohttp. The TCPConnector limits the number of connections to 100 by default. You can override it by creating a TCPConnector with a different limit and passing it to the session.

user4815162342
  • 141,790
  • 18
  • 296
  • 355