14

I am using asyncio for a network framework.

In below code(low_level is our low level function, main block is our program entry, user_func is user-defined function):

import asyncio

loop = asyncio.get_event_loop()
""":type :asyncio.AbstractEventLoop"""


def low_level():
    yield from asyncio.sleep(2)


def user_func():
    yield from low_level()


if __name__ == '__main__':
    co = user_func()
    loop.run_until_complete(co)

I want wrap the low_level as normal function rather than coroutine(for compatibility etc.), but low_level is in event loop. How can wrap it as a normal function?

buhtz
  • 10,774
  • 18
  • 76
  • 149
Robert Lu
  • 1,499
  • 13
  • 22
  • What do you mean by running it as a normal function? Do you want to be able to call it from code that *isn't* running as part of the event loop? – dano Aug 14 '14 at 04:11
  • @dano As I comment "I write web framework, the framework run a event loop, and user function call low level function provide by the web framework. Consider compatibility with other framework, user function may call low level function as normal function rather than a coroutine. Is it impossible to keep compatibility with other framework?" – Robert Lu Aug 14 '14 at 09:08
  • See also https://stackoverflow.com/q/30155138/320911 – Arto Bendiken Jun 21 '17 at 17:16

1 Answers1

20

Because low_level is a coroutine, it can only be used by running an asyncio event loop. If you want to be able to call it from synchronous code that isn't running an event loop, you have to provide a wrapper that actually launches an event loop and runs the coroutine until completion:

def sync_low_level():
    loop = asyncio.get_event_loop()
    loop.run_until_complete(low_level())

If you want to be able to call low_level() from a function that is part of the running event loop, have it block for two seconds, but not have to use yield from, the answer is that you can't. The event loop is single-threaded; whenever execution is inside one of your functions, the event loop is blocked. No other events or callbacks can be processed. The only ways for a function running in the event loop to give control back to the event loop are to 1) return 2) use yield from. The asyncio.sleep call in low_level will never be able to complete unless you do one those two things.

Now, I suppose you could create an entirely new event loop, and use that to run the sleep synchronously from a coroutine running as part of the default event loop:

import asyncio

loop = asyncio.get_event_loop()

@asyncio.coroutine
def low_level(loop=None):
    yield from asyncio.sleep(2, loop=loop)


def sync_low_level():
    new_loop = asyncio.new_event_loop()
    new_loop.run_until_complete(low_level(loop=new_loop))

@asyncio.coroutine
def user_func():
    sync_low_level()

if __name__ == "__main__":
    loop.run_until_complete(user_func())

But I'm really not sure why you'd want to do that.

If you just want to be able to make low_level act like a method returning a Future, so you can attach callbacks, etc. to it, just wrap it in asyncio.async():

loop = asyncio.get_event_loop()

def sleep_done(fut):
    print("Done sleeping")
    loop.stop()

@asyncio.coroutine
def low_level(loop=None):
    yield from asyncio.sleep(2, loop=loop)

def user_func():
    fut = asyncio.async(low_level())
    fut.add_done_callback(sleep_done)

if __name__ == "__main__":
    loop.call_soon(user_func)
    loop.run_forever()

Output:

<2 second delay>
"Done sleeping"

Also, in your example code, you should use the @asyncio.coroutine decorator for both low_level and user_func, as stated in the asyncio docs:

A coroutine is a generator that follows certain conventions. For documentation purposes, all coroutines should be decorated with @asyncio.coroutine, but this cannot be strictly enforced.

Edit:

Here's how a user from a synchronous web framework could call into your application without blocking other requests:

@asyncio.coroutine
def low_level(loop=None):
    yield from asyncio.sleep(2, loop=loop)

def thr_low_level():
   loop = asyncio.new_event_loop()
   t = threading.Thread(target=loop.run_until_complete, args(low_level(loop=loop),))
   t.start()
   t.join()

If a request being handled by Flask calls thr_low_level, it will block until the request is done, but the GIL should be released for all of the asynchronous I/O going on in low_level, allowing other requests to be handled in separate threads.

dano
  • 91,354
  • 19
  • 222
  • 219
  • More general, if I write web framework, the framework run a event loop, and user function call low level function provide by the web framework. Consider compatibility with other framework, user function may call low level function as normal function rather than a coroutine. Is it impossible to keep compatibility with other framework? – Robert Lu Aug 14 '14 at 09:07
  • @RobertLu So you're wondering how you can call an asyncio coroutine from some web framework that has its own event loop (like [Tornado](http://www.tornadoweb.org/en/stable/), for example)? Is there a particular framework you're interested in? – dano Aug 14 '14 at 14:10
  • @RobertLu Can you share which web framework? Unfortunately I don't think there's a one-size-fits-all(-web-frameworks) answer to this. – dano Aug 14 '14 at 14:51
  • Let's talk about flask. In flask user call `connect` function, but I want use a coroutine to replace it(when this request wait query result, other request can get CPU though asyncio's event loop), and I hope user's legacy code can work normally above asyncio. – Robert Lu Aug 14 '14 at 15:10
  • @RobertLu Flask implements [WSGI](http://legacy.python.org/dev/peps/pep-0333/), which means its a single-threaded, synchronous web framework. It isn't built on an event-loop, so it can't be used with asynchronous frameworks that *are* built on an event loop. Concurrency with flask is achieved by putting it behind a webserver that supports using threaded/greenlet/pre-fork workers to handle each request, rather than via asynchronous I/O. See [here](http://flask.pocoo.org/docs/deploying/) for a list of Flask deployment options. – dano Aug 14 '14 at 15:28
  • @RobertLu For a Flask user to call into your asyncio-based network library without blocking other requests, they would need to do so by spawning a thread that runs the your coroutine synchronously, via `loop.run_until_complete(your_coroutine())`. You could provide wrappers for doing this with your framework. – dano Aug 14 '14 at 15:30
  • @RobertLu I added an example at the end of my answer to show how code running inside Flask could call your asyncio-based code, assuming that they're using a webserver that's using threaded/pre-forked workers to handle concurrent requests. – dano Aug 14 '14 at 15:40
  • I suspect the last example may cause threading issues as msot asyncio objects are not thread safe. You should use `call_soon_threadsafe` instead as illustrated [here](https://stackoverflow.com/a/47843980/5540279) – nirvana-msu Feb 08 '18 at 11:11