PEP 0492 - Python 3.5 async keyword

Question

PEP 0492 adds the async keyword to Python 3.5.

How does Python benefit from the use of this operator? The example that is given for a coroutine is

async def read_data(db):
    data = await db.fetch('SELECT ...')

According to the docs this achieves

suspend[ing] execution of read_data coroutine until db.fetch awaitable completes and returns the result data.

Does this async keyword actually involve creation of new threads or perhaps the use of an existing reserved async thread?

In the event that async does use a reserved thread, is it a single shared thread each in their own?

So in this case, how does Python achieve any benefit out of suspending executions in comparison to a generator? — Paul Thompson, Jul 08 '15 at 11:21
Note that the PEP only aims to make certain patterns easier to code; no new functionality is introduced with the syntax. Co-routines *cooperate* in that they allow switching between tasks at pre-defined points (rather than threads which switch execution between different units at arbitrary points). — Martijn Pieters, Jul 08 '15 at 11:30

Martijn Pieters · Accepted Answer · 2015-09-13T16:05:26.510

47

No, co-routines do not involve any kind of threads. Co-routines allow for cooperative multi-tasking in that each co-routine yields control voluntarily. Threads on the other hand switch between units at arbitrary points.

Up to Python 3.4, it was possible to write co-routines using generators; by using yield or yield from expressions in a function body you create a generator object instead, where code is only executed when you iterate over the generator. Together with additional event loop libraries (such as asyncio) you could write co-routines that would signal to an event loop that they were going to be busy (waiting for I/O perhaps) and that another co-routine could be run in the meantime:

import asyncio
import datetime

@asyncio.coroutine
def display_date(loop):
    end_time = loop.time() + 5.0
    while True:
        print(datetime.datetime.now())
        if (loop.time() + 1.0) >= end_time:
            break
        yield from asyncio.sleep(1)

Every time the above code advances to the yield from asyncio.sleep(1) line, the event loop is free to run a different co-routine, because this routine is not going to do anything for the next second anyway.

Because generators can be used for all sorts of tasks, not just co-routines, and because writing a co-routine using generator syntax can be confusing to new-comers, the PEP introduces new syntax that makes it clearer that you are writing a co-routine.

With the PEP implemented, the above sample could be written instead as:

async def display_date(loop):
    end_time = loop.time() + 5.0
    while True:
        print(datetime.datetime.now())
        if (loop.time() + 1.0) >= end_time:
            break
        await asyncio.sleep(1)

The resulting coroutine object still needs an event loop to drive the co-routines; an event loop would await on each co-routine in turn, which would execute those co-routines that are not currently awaiting for something to complete.

The advantages are that with native support, you can also introduce additional syntax to support asynchronous context managers and iterators. Entering and exiting a context manager, or looping over an iterator then can become more points in your co-routine that signal that other code can run instead because something is waiting again.

edited Sep 13 '15 at 16:05

answered Jul 08 '15 at 11:52

Martijn Pieters

1,048,767
296
4,058
3,343

Any case there's a proper motivation behind not implementing async generators based on AsyncInterator as part of built-in syntax with yields? – Andrey Cizov Nov 05 '15 at 11:04
That's something you'll need to discuss on the Python-ideas list, SO comments are not a good medium for that. – Martijn Pieters Nov 05 '15 at 19:21
@MartijnPieters I'm having a hard time understanding asyncio, still. Given your example, the output is something like `2016-03-04 11:53:48.282701` with a new printed line every 1 second. Wouldn't that mean your program is hung up while `await`-ing? I was expecting to see a 1 second delay upon running it, and then a stream of printed datetimes, since the `await asyncio.sleep(...)` should yield as soon as it realizes it has to wait for "io" (in this case, sleep). IOW, the `while` loop would continue running quickly, while the `await`s would eventually catch up. What am I misunderstanding here? – orokusaki Mar 04 '16 at 17:02
2

@orokusaki: the `await asyncio.sleep(1)` line **immediately** yields as *I'm busy, someone else do something*, each time you tried to advance this co-routine, until a full second has passed. It never *blocks*, it just insta-yields with a special 'waiting' message. Because it doesn't block, the co-routine event loop (which polls all the co-routines to find one that is *not* busy waiting) can run a different co-routine. It can 'poll' this co-routine as often as it likes, but that `await` line won't continue the rest of the code until the 1 second time has passed. – Martijn Pieters Mar 04 '16 at 17:05
@MartijnPieters thanks for the quick reply. Ah, so in this case, the coroutine as a whole (including the loop) is in a 'waiting' state, which means it isn't blocking, but if I wanted to achieve something like I've stated (pretending that asyncio.sleep(1) is equivalent to 1 second of Postgres IO), it would need to set up such that the coroutine is run once per action? IOW, I run the line `loop.run_until_complete(query('SELECT ...'))` 10 times, and assuming the query takes about 1 second, I would see the results of all 10 of them in just over 1 second? – orokusaki Mar 04 '16 at 17:10
1

@orokusaki: a proper awaitable takes care of these details for you; e.g. will in turn poll for results from PostgreSQL and unblock with a 'waiting' signal if no results are available yet. The PEP also defines [awaitable iterators](https://www.python.org/dev/peps/pep-0492/#asynchronous-iterators-and-async-for) which are ideal for SQL queries where there may be I/O delays between results; the iterable then 'awaits' for those results, unblocking in between. – Martijn Pieters Mar 04 '16 at 17:44
Awesome. Thanks again @MartijnPieters – orokusaki Mar 04 '16 at 17:50

PEP 0492 - Python 3.5 async keyword

1 Answers1

Linked

Related