Print subprocess stdout line by line in real time in Django StreamHttpResponse

Question

I wrote an API in Django. It runs a shell script in the server and prints some logs. When the client calls the API in the browser, I want the browser to print the logs line by line. So far, the snippet is like:

from django.http import StreamingHttpResponse
import subprocess
import shlex


def test(request):
    cmd = 'bash test.sh'
    args = shlex.split(cmd)
    proc = subprocess.Popen(
        args,
        cwd='/foo/bar',
        shell=False,
        stdout=subprocess.PIPE,
        stderr=subprocess.STDOUT,
    )
    return StreamingHttpResponse(
        (line for line in iter(proc.stdout.readline, '')),
        content_type="text/plain",
    )

And test.sh is like

#!/bin/bash

echo hello
sleep 1
echo world
sleep 1
echo love
sleep 1
echo and
sleep 1
echo peace
sleep 1

The logs are not printed until bash test.sh is finished. How can I make it print the logs as if test.sh is run in the client?

I'm using Python 2.7 and Django 1.11.29. I know they are old, but I still have to use them now. Thanks for help.

Abdul Aziz Barkat · Answer 1 · 2021-05-13T12:13:05.887

You provide the content to StreamingHttpResponse as:

(line for line in iter(proc.stdout.readline, ''))

Here you make an iterable and at iterate over it at the same time and return a generator. Although this won't really matter and you should be getting real time output. The problem may simply be that the output is too fast to actually see the real time difference or perhaps your configuration is incorrect (See Streaming HTTP response, flushing to the browser), Also another problem can be that you are not using the response as a stream with javascript and instead are downloading it fully before using it. You can try using time.sleep() to see if it is just too fast or instead a configuration problem:

import time


def iter_subproc_with_sleep(proc):
    for line in iter(proc.stdout.readline, ''):
        time.sleep(1)
        yield line

def test(request):
    cmd = 'bash test.sh'
    args = shlex.split(cmd)
    proc = subprocess.Popen(
        args,
        cwd='/foo/bar',
        shell=False,
        stdout=subprocess.PIPE,
        stderr=subprocess.STDOUT,
    )
    return StreamingHttpResponse(
        iter_subproc_with_sleep(proc),
        content_type="text/plain",
    )

@ElpieKay although that was correctly a redundancy, I was mistaken and that shouldn't have caused any issues. I have edited the answer, and you can try the code there or there is also a link to another answer in case it is a configuration problem. — Abdul Aziz Barkat, May 13 '21 at 12:08
Thanks for the update. It does not work. The logs are not printed until the script is done. The logs are printed altogether. It can work for existing data, like downloading a file. In my case, the data is constantly being generated and at the same time it's expected to be passed to the client. It reminds me of something like a message queue. But I don't have any idea on this field. — ElpieKay, May 14 '21 at 02:50

Print subprocess stdout line by line in real time in Django StreamHttpResponse

1 Answers1