Python3 asyncio "Task was destroyed but it is pending" with some specific condition

Question

Here is simplified code, which uses python3 coroutine and sets handler for SIGING and SIGTERM signals for stopping job properly:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import argparse
import asyncio
import signal
import sys


def my_handler(signum, frame):
    print('Stopping')
    asyncio.get_event_loop().stop()
    # Do some staff
    sys.exit()


@asyncio.coroutine
def prob_ip(ip_addr):

    print('Ping ip:%s' % ip_addr)
    proc = yield from asyncio.create_subprocess_exec('ping', '-c', '3', ip_addr)
    ret_code = yield from proc.wait()
    if ret_code != 0:
        print("ip:%s doesn't responding" % ip_addr)
        # Do some staff
        yield from asyncio.sleep(2)
        # Do more staff
        yield from asyncio.sleep(16)


@asyncio.coroutine
def run_probing():

    print('Start probing')
    # Do some staff
    yield from asyncio.sleep(1)

    while True:
        yield from asyncio.wait([prob_ip('192.168.1.3'), prob_ip('192.168.1.2')])
        yield from asyncio.sleep(60)


def main():
    parser = argparse.ArgumentParser()
    parser.description = "Probing ip."
    parser.parse_args()

    signal.signal(signal.SIGINT, my_handler)
    signal.signal(signal.SIGTERM, my_handler)

    asyncio.get_event_loop().run_until_complete(run_probing())


if __name__ == '__main__':
    main()

When i run it via:

python3 test1.py

It stops by Ctrl-C without any warnings. But when I run it via:

python3 -m test1

It prints warning by Ctrl-C:

$ python3 -m test1 
Start probing
Ping ip:192.168.1.2
Ping ip:192.168.1.3
PING 192.168.1.2 (192.168.1.2): 56 data bytes
PING 192.168.1.3 (192.168.1.3): 56 data bytes
^C--- 192.168.1.2 ping statistics ---
--- 192.168.1.3 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
1 packets transmitted, 0 packets received, 100% packet loss
Stopping
Task was destroyed but it is pending!
task: <Task pending coro=<prob_ip() running at /tmp/test1.py:22> wait_for=<Future pending cb=[Task._wakeup()]> cb=[_wait.<locals>._on_completion() at /usr/lib/python3.4/asyncio/tasks.py:394]>
Task was destroyed but it is pending!
task: <Task pending coro=<prob_ip() running at /tmp/test1.py:22> wait_for=<Future pending cb=[Task._wakeup()]> cb=[_wait.<locals>._on_completion() at /usr/lib/python3.4/asyncio/tasks.py:394]>

Same warning I get if I install this script via:

from setuptools import setup

setup(name='some_scripts',
      version='1.0.0.0',
      author='Some Team',
      author_email='team@team.ru',
      url='https://www.todo.ru',
      description='Some scripts',
      packages=['my_package'],
      entry_points={'console_scripts': [
          'test1=my_package.test1:main',
      ]},
      )

My python version is "3.4.2"

You might want to use [loop.add_signal_handler](https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.BaseEventLoop.add_signal_handler), though it's not related to your problem. — Vincent, Nov 04 '15 at 09:08
Yes, thanks, this way looks better in my case. But it doesn't help. I still get same warnings. — willir, Nov 05 '15 at 08:24

score 14 · Accepted Answer · edited May 23 '17 at 12:07

Ok. I think I have figured out how should I stop all tasks.

First of all, as far as I understand. BaseEventLoop.stop() is only to stop BaseEventLoop.run_forever(). So one should cancel all tasks via Future.cancel. To get all tasks you can use Task.all_tasks static method.
After cancellation all tasks asyncio.CancelledError exception will be raised from run_until_complete. So one should catch it, if one doesn't want to print it to stderr.
And also, in some cases, I get this error: TypeError: signal handler must be signal.SIG_IGN, signal.SIG_DFL, or a callable object. I found some topics about this error:
- https://bugs.python.org/issue23548
- https://github.com/buildinspace/peru/issues/98
They all say that it can be fixed by closing loop before exiting application.

So we get this code:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import asyncio
import signal


def my_handler():
    print('Stopping')
    for task in asyncio.Task.all_tasks():
        task.cancel()


@asyncio.coroutine
def do_some(some_args):
    while True:
        print("Do staff with %s" % some_args)
        yield from asyncio.sleep(2)


def main():
    loop = asyncio.get_event_loop()

    loop.add_signal_handler(signal.SIGINT, my_handler)

    try:
        loop.run_until_complete(asyncio.wait([do_some(1), do_some(2)]))
    except asyncio.CancelledError:
        print('Tasks has been canceled')
    finally:
        loop.close()


if __name__ == '__main__':
    main()

It works also with signal.signal. But as Vincent noticed loop.add_signal_handler looks better in this case.

But I am still not sure is this the best way to stop all tasks.

(1) unlike futures, tasks may ignore `.cancel()` (catch `CancelledError`) (2) if tasks need to clean up then you could [`loop.run_until_complete(asyncio.gather(*asyncio.Task.all_tasks()))` before `loop.close()`](http://stackoverflow.com/q/27796294/4279) — jfs, Nov 08 '15 at 18:05
If you don't want the signal handler to cancel all the running tasks, have a look at my answer (edited). — Vincent, Nov 08 '15 at 19:01
@J.F.Sebastian Yes, thank you, i have to wait for unfinished tasks before exit. But (I don't know why) asyncio.gather doesn't work in this case (at list in python3.5.0). asyncio.wait works, so we get such code: https://gist.github.com/willir/b521450b66be6e6b238c. Edit: Also I don't know why I need not to catch asyncio.CancelledError from this (actually second one) loop.run_until_complete. — willir, Nov 09 '15 at 14:30
@willir: (1) my guess, you have wrong arguments (`asyncio.wait()` and `asynio.gather()` have different interface e.g., notice `*` in the call above). (2) if you don't understand how/where/when `CancelledError` may be raised when ask a separate question that focuses on it. — jfs, Nov 09 '15 at 17:13

Vincent · Answer 2 · 2015-11-08T18:57:36.977

Use asyncio.gather instead of asyncio.wait:

Cancellation: if the outer Future is cancelled, all children (that have not completed yet) are also cancelled.

Example:

def handler(future, loop):
    future.cancel()
    loop.stop()

@asyncio.coroutine
def do_some(arg):
    while True:
        print("Do stuff with %s" % arg)
        yield from asyncio.sleep(2)

loop = asyncio.get_event_loop()
future = asyncio.gather(do_some(1), do_some(2))
loop.add_signal_handler(signal.SIGINT, handler, future, loop)
loop.run_forever()

Python3 asyncio "Task was destroyed but it is pending" with some specific condition

2 Answers2