Python 3.7 Non-Blocking Request?

Question

I'd like to do a non-blocking http request in Python 3.7. What I'm trying to do is described well in this SO post, but it doesn't yet have an accepted answer.

Here's my code so far:

import asyncio
from aiohttp import ClientSession

[.....]

async def call_endpoint_async(endpoint, data):
    async with ClientSession() as session, session.post(url=endpoint, data=data) as result:
        response = await result.read()
        print(response)
        return response

class CreateTestScores(APIView):
    permission_classes = (IsAuthenticated,)

    def post(self, request):
        [.....]
        asyncio.run(call_endpoint_async(url, data))
        print('cp #1') # <== `async.io` BLOCKS -- PRINT STATEMENT DOESN'T RUN UNTIL `asyncio.run` RETURNS

What is the correct way to do an Ajax-style non-blocking http request in Python?

Does [this question](https://stackoverflow.com/questions/668257/python-simple-async-download-of-url-content) help you? — rassar, Nov 08 '19 at 02:29
@rassar, thanks for the link. I notice that post is from 10 years ago. I understand that python has improved a lot on async since then. Is that post still a current answer for python 3.7? — VikR, Nov 08 '19 at 07:22

user4815162342 · Accepted Answer · 2019-11-09T18:36:23.197

3

Asyncio makes it easy to make a non-blocking request if your program runs in asyncio. For example:

async def doit():
    task = asyncio.create_task(call_endpoint_async(url, data))
    print('cp #1')
    await asyncio.sleep(1)
    print('is it done?', task.done())
    await task
    print('now it is done')

But this requires that the "caller" be async as well. In your case you want the whole asyncio event loop to run in the background, so that. This can be achieved by running it in a separate thread, e.g.:

pool = concurrent.futures.ThreadPoolExecutor()

# ...
    def post(self, request):
        fut = pool.submit(asyncio.run, call_endpoint_async(url, data))
        print('cp #1')

However, in that case you're not getting anything by using asyncio. Since you're using threads anyway, you could as well call a sync function such as requests.get() to begin with.

edited Nov 09 '19 at 18:36

answered Nov 08 '19 at 12:18

user4815162342

141,790
18
296
355

I love threads, but AWS intentionally slows them down on EC2. https://stackoverflow.com/a/17478943/364966. Does the solution you posted rely on threading? – VikR Nov 08 '19 at 17:40
@VikR Thanks, but please note that this is still an unnecessary use of asyncio. – user4815162342 Nov 08 '19 at 22:59
You're very welcome, @user4815162342. Could you please add to your answer to show the best practices approach? Note: when I previously used `import threading [.....] threading.Thread(...` to launch the code (which was in my Django app), it was non-blocking, but the code took about 50 times longer to run than on my local dev system, due to AWS intentionally slowing down Python threads. Here I'm launching the code in an AWS lambda function via AWS API Gateway. – VikR Nov 09 '19 at 00:46
Wait, I think I see what you're saying -- instead of `fut = pool.submit(asyncio.run, call_endpoint_async(url, data))`, call `fut = pool.submit(requests.post(url, data))` -- something like that? If so, is that the correct syntax? – VikR Nov 09 '19 at 01:20
@VikR The syntax would be `pool.submit(requests.post, url, data)`. Since the question was posted under the asyncio tag, it would be nice to actually use asyncio to create a real async request without using threads (that's what asyncio is for!). However, this requires using asyncio for the whole program, and your code is incomplete, so I can't tell whether e.g. `APIView` which you inherit from allows or could be adapted to use async subclasses. – user4815162342 Nov 09 '19 at 08:13
To expand on the above, the approach of calling `asyncio.run()` and using asyncio synchronously is missing out on asyncio's potential. The cide pays the price of setting up and tearing down a whole event loop on every post without providing the actual benefits of asyncio (async code without the use of threads). That approach does work, but is not best practice nor is it how asyncio was designed to be used, which is why I recommended `requests` if using threads. – user4815162342 Nov 09 '19 at 09:34
Fantastic. Thanks @user4815162342! – VikR Nov 09 '19 at 18:30

Python 3.7 Non-Blocking Request?

1 Answers1