Dynamically create asyncio tasks that will run forever

Question

I need to create asyncio tasks (eg: loop.create_task()) dynamically and these tasks will run forever.

What is the best approach of "Fire and forget" in such case ? I am aware that non awaited tasks will be garbage collected, that's the reason I find myself forced to await forever running tasks.

More context:

I have a scheduled routine that needs to create a tasks eventually. As a result, I will have many tasks that will run forever.

Thanks for your help.

user4815162342 · Accepted Answer · 2020-06-19T18:32:51.870

1

What is the best approach of "Fire and forget" in such case ? I am aware that non awaited tasks will be garbage collected,

That is not true: non-awaited tasks created by asyncio.create_task() will not be garbage collected so long as they run. They are owned by the event loop and will run to completion (or never complete, if they are written to run forever) regardless of whether they're being explicitly awaited.

For example, this code creates two "background" tasks without awaiting them, and proceeds to do something else.

async def counter(msg, delay):
    i = 0
    while True:
        print(msg, i)
        i += 1
        await asyncio.sleep(delay)

async def main():
    # create two task
    asyncio.create_task(counter('slow', 1))
    asyncio.create_task(counter('fast', .3))
    # ...and wait forever
    await asyncio.Event().wait()

asyncio.run(main())

that's the reason I find myself forced to await forever running tasks.

It's certainly a good idea to await background tasks when possible, so that you collect their results or receive the exceptions they raise, but it's by no means a requirement. Some tasks don't have a meaningful return value, and it's ok to leave them running on their own.

edited Jun 19 '20 at 18:32

answered Jun 19 '20 at 16:12

user4815162342

141,790
18
296
355

Thanks for your interaction @user4815162342. assuming that we already executed `asyncio.run(main())`, how can we add dynamically more tasks to be awaited in the long run ? (eg: a scheduled routine that creates new tasks). Will `asyncio.run(main())` handle the upcoming tasks ? – OmarZ Jun 19 '20 at 17:43
@OmarZ Tasks can create additional tasks (or cancel existing ones) at any point, the event loop will happily run them all. – user4815162342 Jun 19 '20 at 18:31
I'm still confused as unreferenced tasks will be destroyed by GC. Would you please check https://stackoverflow.com/questions/44304603/python-asyncio-unreferenced-tasks-are-destroyed-by-garbage-collector. @user4815162342 would you please confirm (eg: source) that the created tasks won't ever be GCed ? – OmarZ Jun 20 '20 at 22:52
1

@OmarZ As i demonstranted, with an example, unreferenced tasks will **not** be destroyed. I haven't studied the 3-year-old question in detail, but judging by the answer, the problem was in a buggy awaitable being awaited. That is not an issue with normal code, which you can easily test yourself. – user4815162342 Jun 21 '20 at 21:20

Dynamically create asyncio tasks that will run forever

1 Answers1