Python asyncio ensure_future decorator

Question

Let's assume I'm new to asyncio. I'm using async/await to parallelize my current project, and I've found myself passing all of my coroutines to asyncio.ensure_future. Lots of stuff like this:

coroutine = my_async_fn(*args, **kwargs)
task = asyncio.ensure_future(coroutine)

What I'd really like is for a call to an async function to return an executing task instead of an idle coroutine. I created a decorator to accomplish what I'm trying to do.

def make_task(fn):
    def wrapper(*args, **kwargs):
        return asyncio.ensure_future(fn(*args, **kwargs))
    return wrapper

@make_task
async def my_async_func(*args, **kwargs):
    # usually making a request of some sort
    pass

Does asyncio have a built-in way of doing this I haven't been able to find? Am I using asyncio wrong if I'm lead to this problem to begin with?

your decorator wrapper is blocking. 'def wrapper' should be 'async def wrapper' to not block — Rodrigo Formighieri, May 13 '20 at 05:43
I'm probably not understanding. It's been a while since I've done much with Python. But I think I wanted to synchronously create an asynchronous task. If I made it `async def wrapper`, wouldn't my wrapped function create a coroutine that would create a task when executed? — Brett Beatty, May 13 '20 at 13:59
exactly, so maybe I miss understood your point... I found your question with the will to do exactly what you just said, so I had to add the async word in front of the wrapper in my case. Anyway, for your question purpose, without the 'async' syntax is the way! — Rodrigo Formighieri, May 14 '20 at 00:20

score 4 · Answer 1 · answered Dec 19 '17 at 08:28

4

asyncio had @task decorator in very early pre-released versions but we removed it.

The reason is that decorator has no knowledge what loop to use. asyncio don't instantiate a loop on import, moreover test suite usually creates a new loop per test for sake of test isolation.

answered Dec 19 '17 at 08:28

Andrew Svetlov

16,730
8
66
69

Okay that's interesting to know. Thanks. – Brett Beatty Dec 19 '17 at 19:58

Mikhail Gerasimov · Accepted Answer · 2017-12-19T20:33:13.063

Does asyncio have a built-in way of doing this I haven't been able to find?

No, asyncio doesn't have decorator to cast coroutine-functions into tasks.

Am I using asyncio wrong if I'm lead to this problem to begin with?

It's hard to say without seeing what you're doing, but I think it may happen to be true. While creating tasks is usual operation in asyncio programs I doubt you created this much coroutines that should be tasks always.

Awaiting for coroutine - is a way to "call some function asynchronously", but blocking current execution flow until it finished:

await some()

# you'll reach this line *only* when some() done

Task on the other hand - is a way to "run function in background", it won't block current execution flow:

task = asyncio.ensure_future(some())

# you'll reach this line immediately

When we write asyncio programs we usually need first way since we usually need result of some operation before starting next one:

text = await request(url)

links = parse_links(text)  # we need to reach this line only when we got 'text'

Creating task on the other hand usually means that following further code doesn't depend of task's result. But again it doesn't happening always.

Since ensure_future returns immediately some people try to use it as a way to run some coroutines concurently:

# wrong way to run concurrently:
asyncio.ensure_future(request(url1))
asyncio.ensure_future(request(url2))
asyncio.ensure_future(request(url3))

Correct way to achieve this is to use asyncio.gather:

# correct way to run concurrently:
await asyncio.gather(
    request(url1),
    request(url2),
    request(url3),
)

May be this is what you want?

Upd:

I think using tasks in your case is a good idea. But I don't think you should use decorator: coroutine functionality (to make request) still is a separate part from it's concrete usage detail (it will be used as task). If requests synchronization controlling is separate from their's main functionalities it's also make sense to move synchronization into separate function. I would do something like this:

import asyncio


async def request(i):
    print(f'{i} started')
    await asyncio.sleep(i)
    print(f'{i} finished')
    return i


async def when_ready(conditions, coro_to_start):
    await asyncio.gather(*conditions, return_exceptions=True)
    return await coro_to_start


async def main():
    t = asyncio.ensure_future

    t1 = t(request(1))
    t2 = t(request(2))
    t3 = t(request(3))
    t4 = t(when_ready([t1, t2], request(4)))
    t5 = t(when_ready([t2, t3], request(5)))

    await asyncio.gather(t1, t2, t3, t4, t5)


if __name__ ==  '__main__':
    loop = asyncio.get_event_loop()
    try:
        loop.run_until_complete(main())
    finally:
        loop.run_until_complete(loop.shutdown_asyncgens())
        loop.close()

Thanks, I knew about and am using `asyncio.gather`, but one of my biggest problems with coroutines is that they can only be awaited once. What I'm trying to do is more like this: send 3 requests; when requests 1 & 2 finish, send request 4; when requests 2 & 3 finish, send request 5. I'm just sending tasks 1 & 2 to task 4 and gathering them there, and gathering tasks 2 & 3 in task 5. Is there a way do something like that without tasks? I don't want to have to wait for request 1 to finish before request 5 sends or for request 3 to finish before request 4 sends. — Brett Beatty, Dec 19 '17 at 19:55
Hmm that's an interesting approach. I only need to gather t4 and t5 at the end, though, right? — Brett Beatty, Dec 19 '17 at 20:45
@BrettBeatty yes, this program will work same way if you leave only t4 and t5 in last gather. How exactly better to do depends of how final goal of your script sounds: is it only to request t4 and t5 (which may depend of other requests like 1-3) or it it, for example, all five requests (which should be synchronized in a special way). — Mikhail Gerasimov, Dec 20 '17 at 10:30

Python asyncio ensure_future decorator

2 Answers2