Is it possible to get a result from a coroutine if I'm calling it from a non-asynchronous function and the event loop is already running?

Question

I've been having trouble integrating asyncio into an older codebase. Most of this is manageable, but I am running into troubles with the point where a non-async function needs to invoke a coroutine. This seems to be most easily accomplished by running loop.run_until_complete() on the coroutine in question. When this happens toward the top of the call stack (ie. when we can guarantee that the loop is not already running), it works very well - the coroutine can invoke any other coroutines. Where I'm running into trouble is the case where we cannot guarantee that the loop is not already running. The (admittedly somewhat contrived) code below illustrates this:

import asyncio

import aioredis
from asyncio_extras import (
        contextmanager as async_contextmanager)

async def is_flag_set(redis_pool, key):
    async with acquire_redis_connection(redis_pool) as redis_connection:
        return await redis_connection.get(key)
###

async def helper_1():
    pool = await create_redis_pool()
    return await is_flag_set(pool, 'my_key')

def test_1():
    loop = asyncio.get_event_loop()
    return loop.run_until_complete(
            helper_1())

###

def helper_2(pool=None):
    loop = asyncio.get_event_loop()
    if pool is None:
        pool = loop.run_until_complete(create_redis_pool(
            db_number=0))
    return loop.run_until_complete(is_flag_set(pool, 'my_key'))

def test_2():
    return helper_2()

###

async def helper_3(db_number):
    pool = await create_redis_pool(
            db_number=db_number)
    return helper_2()


def test_3():
    loop = asyncio.get_event_loop()
    return loop.run_until_complete(
            helper_3(db_number=1))

###

# Not relevant to the question, just included
# for completeness:

@async_contextmanager.async_contextmanager
async def acquire_redis_connection(pool):
    connection = await pool.acquire()
    try:
        yield connection
    finally:
        pool.release(connection)

async def create_redis_pool(db_number=0):
    global redis_pool
    redis_pool = await aioredis.create_pool(
            address=('localhost', 6379),
            db=db_number,
            encoding='utf-8',
            minsize=5,
            maxsize=15)
    return redis_pool

if __name__ == '__main__':
    print(test_3())

The point of my question is the function helper_2. When it is called in test_2, the loop isn't running, so it can issue its coroutines safely. When it's called in test_3, though, the loop has already been started, and we get this exception:

  File "/usr/local/lib/python3.6/asyncio/base_events.py", line 408, in run_forever
    raise RuntimeError('This event loop is already running')
RuntimeError: This event loop is already running

I understand why we're getting the exception, but I'm wondering if there are good strategies for managing this problem. I'm not encountering it often in this integration, but I have encountered it a few times - mostly while attempting to write test cases. As far as I can tell from the documentation and from similar questions on StackOverflow, there is no way to "get an answer" from a coroutine if you know that you are in a non-asynchronous function and that the event loop is already running.

Answer 1

there is no way to "get an answer" from a coroutine if you know that you are in a non-asynchronous function and that the event loop is already running.

You can run coroutine in a different event loop. It can be done this way:

global_loop = asyncio.get_event_loop()
# Change current event loop:
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
try:
    return loop.run_until_complete(coro())
finally:
    # return old state to not affect outer code:
    asyncio.set_event_loop(global_loop)

But code above is blocking: it's can't be run concurrently alongside with outer coroutines. It means that you wouldn't get any asynchronous benefit other then achieved inside of coro. Any asynchronous benefit can be only achieved if multiple coroutines run concurrenly inside single event loop. You should understand it clearly.

Note that just the fact you're going to call coroutine in a blocking way while outer event loop is running means that you're risking to freeze this outer event loop. Read more about it here.