python asyncio exceptions raised from loop.create_task()

Question

I want my code to use python logging to log exceptions. In my usual code using await, exceptions are raised normally, so:

try: await code_that_can_raise() except Exception as e: logger.exception("Exception happended")

Works fine.

However, when using loop.create_task(coro())

I'm not sure how can I catch the exception here.
Wrapping the create_task() call obviously won't work. What is the best solution to log every exception in the code?

Answer 1

What is the best solution to log every exception in the code?

If you control the invocation of create_task, but don't control the code in the coro(), then you can write a logging wrapper:

async def log_exceptions(awaitable):
    try:
        return await awaitable
    except Exception:
        logger.exception("Unhandled exception")

then you can call loop.create_task(log_exceptions(coro())).

If you can't or don't want to wrap every create_task, you can call loop.set_exception_handler, setting the exception to your own function that will log the exception as you see fit.

Answer 2

Just so that it has been mentioned: asyncio.Task objects have the methods result and exception.
result:

[...] if the coroutine raised an exception, that exception is re-raised [...]

exception:

[...] If the wrapped coroutine raised an exception that exception is returned [...]

Given a simple setup (in Python 3.7 syntax):

import asyncio
tasks =[]

async def bad_test():
    raise ValueError

async def good_test():
    return

async def main():
    tasks.append(asyncio.create_task(bad_test()))
    tasks.append(asyncio.create_task(good_test()))

asyncio.run(main())

Using result, one could do:

for t in tasks:
    try:
        f = t.result()
    except ValueError as e:
        logger.exception("we're all doomed")

Or, using exception:

for t in tasks:
    if isinstance(t.exception(), Exception):
        logger.exception("apocalypse now")

However, both methods require the Task to be done, otherwise:

If the Task has been cancelled, this method raises a CancelledError exception.

(result): If the Task’s result isn’t yet available, this method raises a InvalidStateError exception.

(exception): If the Task isn’t done yet, this method raises an InvalidStateError exception.

So, unlike the proposal in the other answer, the logging will not happen when the exceptions raise in the tasks, but rather when the tasks are evaluated after they completed.

Answer 3

Expanding on @user4815162342's solution, I created a wrapper around log_exceptions to avoid having to nest every coroutine inside 2 functions:

import asyncio
from typing import Awaitable

def create_task_log_exception(awaitable: Awaitable) -> asyncio.Task:
    async def _log_exception(awaitable):
        try:
            return await awaitable
        except Exception as e:
            logger.exception(e)
    return asyncio.create_task(_log_exception(awaitable))

Usage:

create_task_log_exception(coroutine())

Answer 4

The proper way is to use create_task but you need await it if you want to catch the exception at some point:

import asyncio

async def sleepError(x):
  await asyncio.sleep(x)
  print(1)
  throw_error = 1 / 0

async def sleepOk(x):
  await asyncio.sleep(x)
  print(2)

async def main():
  x = asyncio.create_task(sleepError(1))
  await sleepOk(2)
  
  """
  await x
  # print(3) bellow works without "await x", and print(1) from sleepError as well
  # You can try/except the "await x" line
  # if you "await x" without try/except, print(3) is not executed but error happens.
  # if you don't "await x" you get warning: Task exception was never retrieved
  """
  print(3)


asyncio.run(main())

Answer 5

If you want to react to the exception of a task as soon as it occurs you can use add_done_callback() ( https://docs.python.org/3/library/asyncio-future.html#asyncio.Future.add_done_callback )

asyncio.Task objects is an asyncio.Future like object and has the add_done_callback() method.

In the callback function you just have to get the result() of the future to provoke an exception. with try except you can add custom handling / logging whatever.

import asyncio

async def sleepError(x):
  await asyncio.sleep(x)
  print(1)
  throw_error = 1 / 0

def done_callback(futr):
    try:
        rslt = futr.result()
    except Exception as exc:
        # do_something_with(exc) if you want to (like logging)
        # or just raise
        raise

async def sleepOk(x):
  await asyncio.sleep(x)
  print(2)

async def main():
  x = asyncio.create_task(sleepError(1))
  # without next line exception will only occur at end of main
  x.add_done_callback(done_callback)
  await sleepOk(2)
  print(3)

asyncio.run(main())

If you just want to see the exception on the console, then following callback is sufficient:

def done_callback(futr):
    rslt = futr.result()