Using asyncio for Non-async Functions in Python?

Question

Suppose there is a library that makes various database queries:

import time

def queryFoo():
    time.sleep(4)
    return "foo"

def queryBar():
    time.sleep(4)
    return "bar"

I want to execute those 2 queries concurrently without having to add async to the method signature or adding a decorator. These functions should not depend on asyncio at all.

What is the best way to utilize those non-async functions within asyncio?

I am looking for something of the form:

#I need an 'asyncWrapper'

results = asyncio.gather(asyncWrapper(queryFoo()), asyncWrapper(queryBar()))

Thank you in advance for your consideration and response.

score 18 · Accepted Answer · answered Jun 26 '18 at 22:07

If some function is blocking and not async by nature, only proper way to run it inside asyncio event loop is to run it inside thread using run_in_executor:

# Our example blocking functions
import time


def queryFoo():
    time.sleep(3)
    return 'foo'


def queryBar():
    time.sleep(3)
    return 'bar'


# Run them using asyncio
import asyncio
from concurrent.futures import ThreadPoolExecutor


_executor = ThreadPoolExecutor(10)


async def in_thread(func):
    loop = asyncio.get_event_loop()
    return await loop.run_in_executor(_executor, func)


async def main():
    results = await asyncio.gather(
        in_thread(queryFoo), 
        in_thread(queryBar),
    )

    print(results)


if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    try:
        loop.run_until_complete(main())
    finally:
        loop.run_until_complete(loop.shutdown_asyncgens())
        loop.close()

It does job.

If you however want to avoid using threads only way to do it - is to rewrite queryFoo/queryBar to be async by nature.

Coming from [here](https://stackoverflow.com/questions/55645527/python-asyncio-runtimeeror-this-eventloop-is-already-running/55651314?noredirect=1#comment105248777_55651314), the example you showed above will still need async function for `run_in_executor()`, while I want to make sure the async function converted to sync such that I don't have nested async calls that have nested event loops. One of the solutions I can think of is to spawn new event loop for the inner async calls and block until its result got returned, and continue to next outer loops. Will that work? — Emerson Xu, Jan 01 '20 at 05:28
@EmersonXu *1)* if you want to cast some async function to sync, you just have to do `loop.run_until_complete(asyncfunc())` inside sync funtions. But once you did it, you've done with async world at all: from now on all your outer code is sync. *2)* If you want to cast some sync function to async, use `await loop.run_in_executor(_executor, syncfunc)` inside other async functions. *3)* If you want to do something else, I'm afraid it's hard to me to understand it without reproducible code example. — Mikhail Gerasimov, Jan 03 '20 at 03:40

score 14 · Answer 2 · answered Jun 26 '18 at 21:35

I presume you are after concurrency and hopefully do not insist on using asyncio module itself in which case this little example could be helpful:

import asyncio
import time
from concurrent.futures import ThreadPoolExecutor

def queryFoo():
    time.sleep(2)
    return "FOO"

def queryBar():
    time.sleep(4)
    return "BAR"

with ThreadPoolExecutor(max_workers=2) as executor:
    foo = executor.submit(queryFoo)
    bar = executor.submit(queryBar)
    results = [foo.result(), bar.result()]

print(results)

It runs both queryFoo() and queryBar() in parallel and collects their results in a list in order in which they've been mentioned in an assignment to results.

Looks like he updated the question to say he does need it to be an "asyncio wrapper". — Gabriel G., Apr 11 '21 at 23:54

Using asyncio for Non-async Functions in Python?

2 Answers2

Linked