4

I am designing an application where I create 10000 tasks.

I have a coroutine (async function that queried webservice) and I do asyncio create_task on the coroutine 10000 times with different parameters.

Is there an upper limit of tasks that can be run concurrently?

InfoLearner
  • 14,952
  • 20
  • 76
  • 124
  • There is no fixed limit, and asyncio will try its best to use algorithms with good `O(n)` performance, so it degrades gracefully. At some point you would likely experience GC pauses and other issues that are hard to remove, but 10 thousand tasks is well within the numbers that asyncio was designed to be used with (the "10k connection" issue was a well known problem at the time it was written). See also [this question](https://stackoverflow.com/questions/55761652/what-is-the-overhead-of-an-asyncio-task) for a discussion of memory usage. – user4815162342 Nov 19 '20 at 09:20

1 Answers1

-4

There is no fixed upper limit, It depends on hardware as you're (probably) not using a theoretical computer but a physical hardware one, so you have limited resources.

Typically, the number of threads the run truly concurrently is determined by the number of CPUs and CPU cores (including hyper threading) you have. That is to say that at any given time the number of threads running (in the operating system) is equal to the number of "cores".

Wasif
  • 14,755
  • 3
  • 14
  • 34
  • 5
    The "number of threads" reference is completely irrelevant to asyncio which doesn't use threads at all, and to Python in general, which serializes almost all execution through [GIL](https://wiki.python.org/moin/GlobalInterpreterLock). – user4815162342 Nov 19 '20 at 09:17
  • Unfortunately CPU cores/threads is irrelevant here. – InfoLearner Nov 19 '20 at 10:05