asyncio.future with callback causes Exception after cancellation

Question

I have an awkward problem with several scheduled asyncio.futures (scheduled as message writer of a server) when cancelling them on a graceful shutdown. Each active client connection has a session of active writers.

writer = asyncio.gather(
   *[self._dispatchMessage(message, c) for c in connections],
   loop=self.loop,
   return_exceptions=True
   )

Each writer future (GatheringFuture) gets stored in a list of active sessions...

connection.sessions = set()

...from which the writer future gets removed once it finished. This is achieved by adding a "done_callback" to the future.

writer.add_done_callback(lambda task: connection.sessions.remove(task))
connection.sessions.add(writer)

Once I gracefully shutdown my server, I loop over the active connections and their sessions and cancel (future.cancel()) them to not end up with pending writer futures.

async def cancel_sessions(connection):
    for s in connection.sessions:
    s.cancel()

asyncio.wait(
    [await cancel_sessions(c) for c in client_connections],
    timeout=None
    )

This works, but as soon as I have more connections I keep getting asyncio loop exceptions despite catching exceptions greedily for debugging nearly everywhere. For me it looks, that the problem is caused by the "done_callbacks" added to the writer futures (GatheringFuture). It seems I have to remove the call_backs, otherwise I end up with:

Traceback (most recent call last):
  File "uvloop/cbhandles.pyx", line 49, in uvloop.loop.Handle._run
  File "...server.py", line 353, in <lambda>
    writer.add_done_callback(lambda task: connection.sessions.remove(task))
myUtilsAsyncLoopException: Async exception "<_GatheringFuture finished result=[None, None]>"

If I omit this line from my code...

writer.add_done_callback(lambda task: connection.sessions.remove(task))

...I do not get the problems. My question now is how to handle done_callbacks on cancelled asyncio.futures. My impression was, that I don't have to manually remove such callbacks from a future before cancelling it. But it seems that when cancelling a future with a callback, these callbacks might throw exceptions when their related future is cancelled.

I also don't yet understand why this happens only when I have more than 2 connections because I see no big difference in handling them.

score 0 · Answer 1 · answered Mar 31 '18 at 18:52

0

I guess I solved it myself. It often helps to formulate a problem in a question to understand the situation better. My problem seems not to be connected with asyncio, but a simple logical problem. All connection.sessions got the same (writer) future and indeed asyncio.future "done_callbacks" were still called on cancellation but adding the same future (writer) to several connection.sessions caused a key error in the lambda callback.

answered Mar 31 '18 at 18:52

geotom

21
5

This is a classic Python gotcha (and not only Python, the same issue exists in JavaScript and many other languages); see https://stackoverflow.com/questions/7546285/creating-lambda-inside-a-loop?lq=1 for idiomatic solutions. – user4815162342 Mar 31 '18 at 22:23
1

The issue seems to be that not only lambdas were affected, but also when using normal functions as closures. The looped variable was never bound. Indeed a classic... – geotom Apr 03 '18 at 10:54
Even worse, the loop variable is bound, but it's bound lazily. In a sense, it is not a bug - since the `for` loop actually changes the value of the variable, the closure gets access to the new value. In situations that don't involve a `for` loop you actually _want_ the current behavior - assigning to the variable is typically done precisely so that the closure can pick up its new value. This is why the issue was never resolved, it is impossible to do so without either changing the meaning of `for` or changing the semantics of closures, neither for the better. – user4815162342 Apr 03 '18 at 11:04

score 0 · Accepted Answer · answered Mar 31 '18 at 23:02

For everyone reading this question: The problem originated from registering "done callbacks" to a future for each available connection. This happened in a for loop, but the callback was never bound to the actual connection when the future completed but to the last connection of the for loop. Thus always the same connection object.

The documentation advises to use functools to bind certain values to a future's callback. This finally worked (binding values in a for loop to the callback) and had no "funny" side-effects.

asyncio.future with callback causes Exception after cancellation

2 Answers2