Django Redis-Queue pass result from background job into html

Question

My small Django project only has one view. The user can enter some options and press submit, hence commiting the job that takes >3 minutes. I use redis-queue to carry out the jobs via a background process - the web process simply returns (and renders the .html). All this works. Question is, how can I pass the background's result (filename of file do be downloaded) after completion back into the html so I can display a download button?

I guess I would have to put a small Java / jQuery script into the .html that keeps polling if the job has completed or not (perhaps making use of RQ's on_success / on_failure functions?). However, I'm unsure about the approach. Searching the matter has brought no success (simular issue). Could anyone guide me as to what's the correct way of doing this? I do not intend to use Celery. Ideal solution would be the one providing some code or a clear path. Here's a stripped example:

view.py

from django.shortcuts import render
from django.contrib import messages
from django.views.decorators.csrf import csrf_exempt
from django.http import HttpResponse
from .forms import ScriptParamForm
from .queue_job import my_job

@csrf_exempt
def home(request):
    context = {}

    if request.method == 'POST':

        if request.POST.get("btn_submit"):
            req_dict = dict(request.POST)         
            form     = ScriptParamForm(request.POST)
            context['form'] = form

            if form.is_valid():
                req_dict = dict(request.POST)
                job = my_job(outdir=tmp_dir, **req_dict)
                context['job'] = job
            return render(request, 'webapp/home.html', context)

        elif request.POST.get("btn_download"):
            serve_file = request.POST.get("btn_download")

            with open(serve_file, 'rb') as fp:
                response = HttpResponse(fp, headers={
                    'Content-Type': 'application/zip',
                    'Content-Disposition': f'attachment; filename="{serve_file}"'
                })
                return response
    else:
        form = ScriptParamForm()
        context['form'] = form
        return render(request, 'webapp/home.html', context)

home.html

{% extends 'base.html' %}
{% block content %}

<form method="POST" autocomplete="on">
    {% csrf_token %}
     
    <!-- FORM FIELDS -->
    ...

    <!-- SUBMIT / DOWNLOAD BUTTONS -->
    <div>
        <button type="submit" name="btn_submit" value="submit" class="btn btn-primary" id="btn-submit">Submit</button>

        {% if serve_file %}
        <button type="submit" name="btn_download" value={{serve_mode}} class="btn btn-primary">Download</button>
        {% endif %}
        
        <span class="not-visible" id="calc-msg"> This may take a few, relax! :) </span>
    </div>

</form>
{% endblock content %}

main.js

var x = document.getElementById("btn-submit")

console.log(x)
x.onclick = function () {
    $("#calc-msg").removeClass("not-visible");
    setInterval(function(){blink()}, 1000);
};

function blink() {
    $("#calc-msg").fadeTo(100, 0.1).fadeTo(200, 1.0);
}

Answer 1

There are several ways with different advantages and disadvantages each.

The simplest approach is exactly as you mentioned: polling. The disadvantages of polling are:

• it's not real time (you'll get your result no sooner than on your next poll request)
• it doesn't scale so well. This is only a problem is you have many clients all waiting for their results at the same time. For example: 100k clients polling once every 5 seconds = 20k requests per second to the server.

The major advantage is the simplicity of implementation.

Alternatives for real-time are long polling, websockets, and webrtc data channels. The first is all but obsolete, the second is what I'd recommend, the last is sort of an abuse of a protocol that exists but isn't intended for this purpose (it works none the less!). Websockets and webrtc both provide 2-way communication between your browser and the server. The server can push a message to your browser which listens to reacts on these events. Due to the synchronous req/res model of Django, it's not capable of using either out of the box so you'd need to look into django-channels or even replace django completely with something like aiohttp or fastapi.

Assuming that you don't need real-time feed back and that you're not serving your app to 10s of thousands of users I'd suggest you stick will polling.

High level overview:

1. Client makes a request.

2. Server creates a "Job" record with a unique identifier and immediately returns this identifier to the client as well as pushing your reqis-queue message. Let's say that a "Job" has 3 possible states: pending / finished / failed.

class Job(models.Model):
    PENDING = 0
    FINISHED = 1
    FAILED = 2
    state = models.IntegerField(default=Job.PENDING)
    result_path = models.CharField(default=None, null=True)

In your view, create an instance and pass the identifier to the queue:

job_instance = Job.objects.create()
req_dict = dict(request.POST)
job = my_job(outdir=tmp_dir, job_id=job_instance.id, **req_dict)
context['job'] = job
context['job_id'] = job_instance.id

3. The client uses this identifier to poll for the results. As long as the Job is still pending it should continue requesting updates to the server.

let myInterval = setInterval(() => {
  fetch('your-url.com/job-poll/{job_id}.json')
    .then(response => response.json())
    .then(data => {
      if (data.job.state === 1) { // finished
          // download {data.job.result_path} and
          clearInterval(myInterval);
      } else if (data.job.state === 2) { // failed
          // tell the user
          clearInterval(myInterval);
      } else { // pending
          // job is still pending, do nothing.
      }
    })
}, 3000)

A new view to handle the polling:

from django.http import JsonResponse

def job_poll(request, job_id):
    j = Job.objects.get(id=job_id)
    return JsonResponse({
      "job": {
        "state": j.state,
        "result_path": j.result_path,
      },
    })

4. When the redis-queue worker is finished, it should update the Job record with the final state: finished or failed

j = Job.objects.get(id=job_id)
j.state = Job.FINISHED
j.result_path = '...'
j.save()

5. Finally on the next poll request, your client will now be aware of the finished job and can now download the result, error message, etc.

(note: all code is pseudo code, nothing was tested here)