Calling coroutine and getting future in asyncio.Protocol.data_received()?

Question

I need to get the future result inside asyncio loop, it is similar to Calling a coroutine from asyncio.Protocol.data_received

But asyncio in PY35 and PY34 are completely different, here are the code which could run correctly in PY34, but in PY35 it will pause at yield from and never return.

# PY34
class RelayClient(asyncio.Protocol):
    pass

class Server(asyncio.Protocol):
    def data_received(self, data):
        # I need to connect to another address, and get future result at current function.
        # Also I could not run loop.run_until_complete().
        loop = asyncio.get_event_loop()
        result = yield from loop.create_connection(RelayClient, 'www.google.com', 443)
        do_some_thing_with_result(result)

So, how to do this in python 3.5?

Any advice is appreciated.

Possible duplicate of [Calling a coroutine from asyncio.Protocol.data\_received](https://stackoverflow.com/questions/29126340/calling-a-coroutine-from-asyncio-protocol-data-received) — kwarunek, Aug 21 '17 at 05:24
The problem still looks the same. You can't use `await` or `yield from` in `Protocol.data_received`, instead use `ensure_future` and set proper callback. — kwarunek, Aug 21 '17 at 06:18
@kwarunek Thanks for your reply, I know `ensure_fure` with `add_done_callback`, but this will make my code confusing. So is that means set a call back is the only method? — hfawja, Aug 21 '17 at 07:00

score 1 · Accepted Answer · answered Aug 21 '17 at 12:12

You cannot await a coroutine from a function that is not a coroutine. data_received is not a coroutine, so as was mentioned in the comments, you need to use the ensure_future helper to create a "background" task from your coroutine.

No need to start using callbacks however:

async def do_stuff(data):
        result = await loop.create_connection(RelayClient, 'www.google.com', 443)
       await do_some_thing_with_result(result)

class Server(asyncio.Protocol):
    def data_received(self, data):
        asyncio.ensure_future(do_stuff(data))

I would point out however, that asyncio gives no garanties whatsoever that data_received will be called with the complete data you are expecting. Usually the pattern you see in a Protocol looks a lot like this:

async def process_line(line):
    ...

class Server(asyncio.Protocol):
    def __init__(self):
        self.buffer = b''

    def data_received(self, data):
        self.buffer += data

        if b'\n' not in self.buffer:
            return

        line, self.buffer = self.buffer.split(b'\n')
        fut = asyncio.ensure_future(process_line(line))
        fut.add_done_callback(self._handle_exception)

    def _handle_exception(self, fut):
        if fut.exception() is not None:
            print('Processing failed', fut.exception())

(this is just an example, it copies the buffer way too much and would be very inefficient in most production use-cases)

can you give some hint on how you would avoid buffer copies ? — allan.simon, Jan 31 '18 at 15:53
There are different possible approach, and all have their drawbacks. The asyncio framework itself now uses a `buffer = bytearray()`, and calls `buffer.extend(data)` for every chunks it receives to accumulate the data without copying the already-buffered data too much: https://github.com/python/cpython/blob/3.6/Lib/asyncio/streams.py#L424 — Arthur, Feb 01 '18 at 09:44
aiohttp on the other hand uses a `buffer = []` approach, and each new chunk is simply appended to the list (ie nothing is copied when the data is received), but then the chunks have to be eventually copied to be correctly joined: https://github.com/aio-libs/aiohttp/blob/master/aiohttp/streams.py#L284 — Arthur, Feb 01 '18 at 09:47

Calling coroutine and getting future in asyncio.Protocol.data_received()?

1 Answers1