async exec in python

Question

I'd like to call exec in an async function and do something like the following code (which is not valid):

import asyncio

async def f():
    await exec('x = 1\n' 'await asyncio.sleep(x)')

More precisely, I'd like to be able to wait for a future inside the code that runs in exec.

How can this be achieved?

Answer 1

Note: F-strings are only supported in python 3.6+. For older versions, use %s, .format() or the classic + concatenation.

async def aexec(code):
    # Make an async function with the code and `exec` it
    exec(
        f'async def __ex(): ' +
        ''.join(f'\n {l}' for l in code.split('\n'))
    )

    # Get `__ex` from local variables, call it and return the result
    return await locals()['__ex']()

---

Known issues:

• If you use new lines in a string (triple quotes), it will mess up the formatting.

Answer 2

Yours problem is that you are trying to await to None object- exec ignores the return value from its code, and always returns None. If you want to execute and await to the result you should use eval- eval returns the value of the given expression.

Your's code should look like this:

import asyncio

async def f():
    exec('x = 1')
    await eval('asyncio.sleep(x)')

loop = asyncio.get_event_loop()
loop.run_until_complete(f())
loop.close()

Answer 3

This is based off @YouTwitFace's answer, but keeps globals unchanged, handles locals better and passes kwargs. Note multi-line strings still won't keep their formatting. Perhaps you want this?

async def aexec(code, **kwargs):
    # Don't clutter locals
    locs = {}
    # Restore globals later
    globs = globals().copy()
    args = ", ".join(list(kwargs.keys()))
    exec(f"async def func({args}):\n    " + code.replace("\n", "\n    "), {}, locs)
    # Don't expect it to return from the coro.
    result = await locs["func"](**kwargs)
    try:
        globals().clear()
        # Inconsistent state
    finally:
        globals().update(**globs)
    return result

It starts by saving the locals. It declares the function, but with a restricted local namespace so it doesn't touch the stuff declared in the aexec helper. The function is named func and we access the locs dict, containing the result of the exec's locals. The locs["func"] is what we want to execute, so we call it with **kwargs from aexec invocation, which moves these args into the local namespace. Then we await this and store it as result. Finally, we restore locals and return the result.

Warning:

Do not use this if there is any multi-threaded code touching global variables. Go for @YouTwitFace's answer which is simpler and thread-safe, or remove the globals save/restore code

Answer 4

Thanks for all the suggestions. I figured out that this can be done with greenlets along async, since greenlets allow performing "top level await":

import greenlet
import asyncio

class GreenAwait:
    def __init__(self, child):
        self.current = greenlet.getcurrent()
        self.value = None
        self.child = child

    def __call__(self, future):
        self.value = future
        self.current.switch()

    def __iter__(self):
        while self.value is not None:
            yield self.value
            self.value = None
            self.child.switch()

def gexec(code):
    child = greenlet.greenlet(exec)
    gawait = GreenAwait(child)
    child.switch(code, {'gawait': gawait})
    yield from gawait

async def aexec(code):
    green = greenlet.greenlet(gexec)
    gen = green.switch(code)
    for future in gen:
        await future

# modified asyncio example from Python docs
CODE = ('import asyncio\n'
        'import datetime\n'

        'async def display_date():\n'
        '    for i in range(5):\n'
        '        print(datetime.datetime.now())\n'
        '        await asyncio.sleep(1)\n')

def loop():
    loop = asyncio.get_event_loop()
    loop.run_until_complete(aexec(CODE + 'gawait(display_date())'))
    loop.close()

Answer 5

Here is a more robust way using the builtin ast module:

import ast

async def async_exec(stmts, env=None):
    parsed_stmts = ast.parse(stmts)

    fn_name = "_async_exec_f"

    fn = f"async def {fn_name}(): pass"
    parsed_fn = ast.parse(fn)

    for node in parsed_stmts.body:
        ast.increment_lineno(node)

    parsed_fn.body[0].body = parsed_stmts.body
    exec(compile(parsed_fn, filename="<ast>", mode="exec"), env)

    return await eval(f"{fn_name}()", env)

Answer 6

Here's a module using AST to do stuff. This means that mutli-line strings will work perfectly and line-numbers will match the original statements. Also, if anything is an expression, it is returned (as a list if there are multiple, otherwise as just an element)

I made this module (check the revision history of this answer for more details on the inner workings). I use it here

Answer 7

Just use this function:

import asyncio
async def async_exec(code):
    t = [None]
    exec('async def _async_exec():\n return {}\nt[0] = asyncio.ensure_future(_async_exec())'.format(code))
    return await t[0]

Here is a code example that can directly run. (It works for Python 3.6.8)

import asyncio
async def async_exec(code):
    t = [None]
    exec('async def _async_exec():\n return {}\nt[0] = asyncio.ensure_future(_async_exec())'.format(code))
    return await t[0]

async def p(s):
    await asyncio.sleep(s)
    return s


async def main():
    print(await async_exec('await p(0.1) / await p(0.2)'))


asyncio.get_event_loop().run_until_complete(main())

I try to explain it, define an async function in exec. Inside the async function, run the code you want. But exec doesn't have return value, use t[0] to store an asyncio future, await the future outside of exec to get the return value.