Awaiting a non-async function in python

Question

I was looking at some python tutorial for asyncio package. A paradigmatic example is the following:

import asyncio


async def first_function():
    print("Before")
    await asyncio.sleep(1)
    print("After")

async def second_function():
    print("Hello")

async def main():
    await asyncio.gather(first_function(), second_function())


if __name__ == "__main__":
    asyncio.run(main())

Even in more complex examples, the innermost async function one is awaiting is either asyncio.sleep or some function coming from some "async library". But how are these function made of in terms of non-async functions? To me more concrete: suppose I have a non-async function coming from an external library whose execution time is quite long long_function. (This function do not contain any await, it may be written in another language, for example). How can I write an async function wrapper out of it (maybe with a future) so that I can write

import asyncio

async def long_function_wrapper():
     ...
     #something with long_function()
     ...


async def first_function():
    print("Before")
    await long_functin_wrapper()
    print("After")

async def second_function():
    print("Hello")

async def main():
    await asyncio.gather(waiting_function(), short_function())


if __name__ == "__main__":
    asyncio.run(main())

and the behavior is the same as in the former example? (i.e. the order of the output is the same)

Louis Lac · Answer 1 · 2023-03-06T11:36:09.297

All asynchronous primitives (asyncio.sleep, aiohttp.get, etc.) are based on low level concurrency primitives usually provided by the operating system and thus are generally not composed of other nested async function calls.

Those methods rely on threads, locks (mutexes), OS timers, network port listening, etc. In a nutshell, the OS notifies your program when the the task is done. This answer could also be relevant.

There are numerous ways to create an async function, the most obvious one being to compose other async functions. If the function you want to make async is IO-bound (network, file reading, sleep, etc.), you can just wrap in in asyncio.to_thread which will run it in a separate thread:

import asyncio
import requests


def io_bound_task():
  return requests.get("https://some/url.com")


async def main():
  return await asyncio.to_thread(io_bound_task)


if __name__ == "__main__":
  asyncio.run(main())

If your task is CPU-bound (heavy computations), then you will want to offload it to a thread pool (for GIL-releasing tasks) or a process pool (for other tasks):

import asyncio
from concurrent.futures import ProcessPoolExecutor
from math import sqrt


def cpu_bound_task():
  result = 0
  for _ in range(100_000_000):
    result = sqrt(result + 1)
  return result


async def main():
  loop = asyncio.get_running_loop()

  with ProcessPoolExecutor() as p:
    return await loop.run_in_executor(p, cpu_bound_task)


if __name__ == "__main__":
  asyncio.run(main())

score 1 · Answer 2 · answered Mar 04 '23 at 12:27

If you are interested in the actual concepts behind the async/await syntax, I'd refer you to PEP 492. To understand this more deeply, you should familiarize yourself with the ideas behind coroutines and - underlying that - iterators. Lastly, you should understand the purpose of the event loop (and schedulers more broadly) and how that ties into iterators/generators.

It is technically possible to deconstruct existing "regular" functions and turn them into coroutines, but it is not at all trivial. Generally speaking, since the event loop runs in a single thread, unless your long_function is asynchronous (i.e. yields control back to the scheduler at some point), executing it in an async context will simply block the event loop until it is finished.

There are third-party libraries that attempt to do what you describe, i.e. "async-ify" non-async functions. I cannot speak to how well this works in practice and what limitations those have.

If you have no control over that long_function, it is non-async, and you want it to execute concurrently with something else, you might be better off just running it in a separate thread or process (depending on what it does). The distinctions between these three built-in concurrency options in Python have been discussed at length on this site and elsewhere.

Here is a sample:

multiprocessing vs multithreading vs asyncio

Awaiting a non-async function in python

2 Answers2