How can I run an async function from a django view to run in the background while I process requests in the main loop?

Question

As I said in the title, I want a task to run in the background processing a large database query while I handle requests from my frontend. Can I even do that with async/asyncio? I was told it was possible...

For the purpose of context, I would like to do something like the following. Note also that I don't really need the function to tell me when it's done (though I'd sure like to know if it's possible) since I just check if the .json file was finally written.


def post_data_view(request):
    if request.method == 'POST':
        ...
        do_long_query_in_the_background(some_data)
        return HttpResponse('Received. Ask me in a while if I finished.')

def is_it_done_view(request):
    if request.method == 'GET':
        data = find_json()
        if data:
            return JsonResponse(data)
        else:
            return HttpResponse('Not yet dude...')

async def do_long_query_in_the_background(data):
    # do some long processing...
    # dump the result to a result.json
    return

I was told this was possible with async but I really find it hard to understand. For context, I tried to simplify this a lot, and even then I found I didn't quite understand what was happening:


async def f():
    while True:
        print(0)
        await asyncio.sleep(2)

asyncio.create_task(f())

even this code I tried fails sys:1: RuntimeWarning: coroutine 'f' was never awaited, but it does work on console and I do not understand why that is.

I also wondered if this is at all possible and safe to do with threading maybe?

I'm extremely frustrated with this because the general solution suggested in other threads seems to be to just use celery but it really feels like an overkill for a not so complex problem.

sounds like you don't have an event loop running. have you created an event loop? — el_oso, Mar 26 '21 at 09:59

score 1 · Answer 1 · edited May 11 '23 at 09:21

import asyncio

async def f():
    while True:
        print(0)
        await asyncio.sleep(2)

asyncio.run(f())

the run method from asyncio sets up your event loop and creates the Task object, schedules it to run and waits for it to complete before executing any other code.

While appearing quite simple, async programming requires a very different approach to programming as things can happen in any order and you have to think very carefully about what order is important for functions to complete.

For your use case though, you might be able to use threading. You might be able to create a new thread any have it running in the background. There is a performance penalty due to switching between threads but your user may have a better experience if most of the processing is done server side.

#this runs forever as your loop never terminates
from threading import Thread
from time import sleep

def f():
    while True:
        print(0)
        await asyncio.sleep(2)

def main():
     print('starting new thread...')
     t = Thread(target=f)
     t.start()
     print('continuing other tasks...')
     sleep(5)
     print('still more things to do...')

The example using `Thread` does not compile as `f` is not `async def`. I think https://stackoverflow.com/a/75310136/3936440 goes into the right direction. — ViRuSTriNiTy, Mar 11 '23 at 13:11

score 0 · Answer 2 · answered Apr 11 '21 at 07:40

You need to define your view function with aync . In your situation it should be: async def post_data_view(request):
In your async view function: Firstly need to get loop. Then create a task in the loop. It should be:

    async def post_data_view(request):
        if request.method == 'POST':
        ....
        loop = asyncio.get_event_loop()
        loop.create_task(do_long_query_in_the_background(some_data))
        return HttpResponse('Received. Ask me in a while if I finished.')

Lane Henslee · Answer 3 · 2023-03-13T09:23:19.550

I have the perfect solution for you. Exactly what you want. This solution works with daphne since it creates a global loop that you can create new tasks in without the need to start a new loop.

from daphne.server import twisted_loop

# heres a view that you want to start a background function in
def hello_background(request):
    twisted_loop.create_task(long_task())

This will run the task in the background by adding an event to the global loop.

How can I run an async function from a django view to run in the background while I process requests in the main loop?

3 Answers3