0

I have a pool of tasks in a list and each of those tasks take a different amount of time to complete. To mimick this I'll use this piece of code: tasks = [asyncio.create_task(asyncio.sleep(i)) for i in range(10)]

If I use the asyncio gather API like this: await asyncio.gather(*tasks), then this statement blocks the event loop untill all the tasks in the current batch finishes.


The problem however is in my program, I want to execute those tasks with a ThreadPoolExecutor like interface with x number of workers. This will allow for executing tasks more efficiently (Tasks which finish up early will be replaced by another task instead of waiting for all tasks of that batch to complete).

How can I achieve this using python asyncio?

Shakir
  • 270
  • 2
  • 11
  • Does this answer your question? [How to properly create and run concurrent tasks using python's asyncio module?](https://stackoverflow.com/questions/29269370/how-to-properly-create-and-run-concurrent-tasks-using-pythons-asyncio-module) – ti7 Nov 17 '22 at 15:32
  • if you have many jobs to process which are [compute-bound](https://en.wikipedia.org/wiki/CPU-bound), asyncio may not help you, and instead a threadpool or similar reading jobs from a queue might be a better design – ti7 Nov 17 '22 at 15:33
  • The tasks are network IO bound so that's why I chose asyncio over threading and multiprocessing. – Shakir Nov 18 '22 at 11:33
  • Also the question you linked in the above comment unfortunately is not what I was looking for. – Shakir Nov 18 '22 at 11:41

1 Answers1

1

asyncio.gather() waits for the tasks to finish. If you don’t want to wait, just don’t call asyncio.gather().

The tasks will kick off even if you don’t call gather(), as long as your event loop keeps running.

To keep an event loop running, call loop.run_forever() as your entry point, or call asyncio.run(coro()) and in coro() call await asyncio.Future() to “wait forever”.

You may also check out the TaskGroup class of Python 3.11 to see if it’s semantics meet what you need.

fancidev
  • 144
  • 1
  • 4
  • Thank you, this has partially answered my question as I did not know that tasks start automatically once created in background.However, I'm still looking for (maybe) a library which works similar to ThreadPoolExecutor / ProcessPoolExecutor but for async functions. – Shakir Nov 23 '22 at 17:59