calling sync functions from async function

Question

I am in the process of trying to port a flask app to quart to utilise asyncio. I don't think my current approach is working, as my entire function chain is written without async in mind - consider the following:

def long_running_task(task):
    result = some_synchronous_function(task)
    return result

@app.route('/<task>', methods=['GET'])
async def do_task(task):
    ok = await long_running_task(task)
    if ok:
        return (ok.result)
    else:
        return ('Something went wrong')

If long_running_task and its whole chain of function calls are not declared as async, am I actually getting any benefit from my route being declared as async?

Nope, that shouldn't even work. You can only `await` awaitables — async functions etc. To use the power of async, you need to transform your I/O (DB access, etc.) to asynchronous access. — L3viathan, Feb 14 '19 at 07:34

J. Taylor · Accepted Answer · 2019-02-14T09:51:27.057

28

To run a blocking synchronous function from asyncio, without blocking the main event loop, you can use loop.run_in_executor() to run the blocking function in a ThreadPoolExecutor or ProcessPoolExecutor` (i.e. in its own thread or process).

From within the async function you want to call it from:

loop = asyncio.get_event_loop()

result = await loop.run_in_executor(None, long_running_task, task)

The first argument None is to tell it to use the default executor for the loop. Obviously do_task() will still have to wait for result to complete, but while it is waiting, other async tasks will be able to run in event-loop.

edited Feb 14 '19 at 09:51

answered Feb 14 '19 at 07:37

J. Taylor

4,567
3
35
55

Is it possible to call multiple functions to run concurrently in the same `loop.run_in_executor`? – y_159 Sep 10 '21 at 18:53
@y_159 check out `asyncio.gather()`, you can use it to run them in parallel. – Leah Sapan Oct 07 '21 at 19:06
@LukeSapan `asyncio.gather()` is used to run tasks created by `asyncio..create_task()` concurrently. How would i use that here for `loop.run_in_executor()`. I think it starts executing the blocking function without creating a task object? My question is how to use a single `loop.run_in_executor()` to run multiple blocking functions concurrently. – y_159 Oct 08 '21 at 06:06
1

@y_159 You can invoke `loop.run_in_executor` multiple times _within_ `asyncio.gather(...)`. Even though you're calling `loop.run_in_executor` multiple times, they'll all be using the default shared executor, so you aren't spawning new executors each time. Alternatively, if you don't want to use the default executor, you can create your own shared one, and pass it in as the first parameter to each call (instead of `None` as shown in the example above). – Leah Sapan Oct 22 '21 at 01:17

calling sync functions from async function

1 Answers1

Linked