4

I'm evaluating different patterns for periodic execution (actual sleep/delays ommited for brevity) using the Python 3 asyncio framework, and I have two pieces of code that behave diffrently and I can't explain why. The first version, which uses yield from to call itself recursively exhausts the stack in about 1000 iterations, as I expected. The second version calls the coroutine recursively, but delegates actual event loop execution to asyncio.async and doesn't exhaust the stack. Can you explain in detail why the stack isn't being used by the second version? What are the differences between the two ways of executing this coroutine?

First version (yield from):

@asyncio.coroutine
def call_self(self, i):
    print('calling self', i)
    yield from self.call_self(i + 1)

Second version (asyncio.async):

@asyncio.coroutine
def call_self(self, i):
    print('calling self', i)
    asyncio.async(self.call_self(i + 1))
dano
  • 91,354
  • 19
  • 222
  • 219
Ivailo Karamanolev
  • 799
  • 7
  • 20
  • have you tried to run it? The second version prints only couple of times (I would expect that it prints only once if it is run as `loop.run_until_complete(self.call_self(0))`) – jfs Oct 24 '14 at 08:00
  • @J.F.Sebastian I have indeed tried to run it and I don't run it with run_until_complete, but run_forever. Both of these try to print infinitely with run_forever, but the first one exhausts the stack, while the second one runs forever (2 million+). – Ivailo Karamanolev Oct 24 '14 at 08:40
  • then try to run both variants using `run_until_complete()`: it demonstrates the difference between waiting for the results and just scheduling coroutines. – jfs Oct 24 '14 at 09:33

1 Answers1

11

The first example, using yield from, actually blocks each instance of call_self until its recursive call to call_self returns. This means the call stack keeps growing until you run out of stack space. As you mentioned, this is the obvious behavior.

The second example, using asyncio.async, doesn't block anywhere. So, each instance of call_self immediately exits after running asyncio.async(...), which means the stack doesn't grow infinitely, which means you don't exhaust the stack. Instead, asyncio.async schedules call_self gets to be executed on the iteration of the event loop, by wrapping it in a asyncio.Task.

Here's the __init__ for Task:

def __init__(self, coro, *, loop=None):
    assert iscoroutine(coro), repr(coro)  # Not a coroutine function!
    super().__init__(loop=loop)
    self._coro = iter(coro)  # Use the iterator just in case.
    self._fut_waiter = None
    self._must_cancel = False
    self._loop.call_soon(self._step)  # This schedules the coroutine to be run
    self.__class__._all_tasks.add(self)

The call to self._loop.call_soon(self._step) is what actually makes the coroutine execute. Because it's happening in a non-blocking way, the call stack from call_self never grows beyond the call to the Task constructor. Then the next instance of call_self gets kicked off by the event loop on its next iteration (which starts as soon as the previous call_self returns, assuming nothing else is running in the event loop), completely outside of the context of the previous call_self instance.

dano
  • 91,354
  • 19
  • 222
  • 219
  • 1
    +1. It is documented that `async()` schedules the coroutine (no need to dig up the source). Important detail: `asyncio.coroutine` decorator converts the `call_self` into a generator otherwise `async(call_self())` would exhaust stack whatever `async()` does. – jfs Oct 27 '14 at 08:30