9

I need a HTML webpage in my Django app to load and show the continous output of a script in a scrollable box. Is this possible?

I'm presently using a subprocess to run a Python script, but the HTML page won't load until after the script has finished (which can take about 5 minutes). I want the users to see something is happening, rather than just a spinning circle.

What I have already also unloads the full output of the script with "\n" in the text; I'd like it to output each new line instead, if possible.

My code is as follows:

Views.py:

def projectprogress(request):
    GenerateProjectConfig(request)
    home = os.getcwd()
    project_id = request.session['projectname']
    staging_folder = home + "/staging/" + project_id + "/"
    output = ""
    os.chdir(staging_folder)
    script = home + '/webscripts/terraformdeploy.py'
    try:
        output = subprocess.check_output(['python', script], shell=True)
    except subprocess.CalledProcessError:
        exit_code, error_msg = output.returncode, output.output
    os.chdir(home)
    return render(request, 'projectprogress.html', locals())

projectprogress.html:

<style>
  div.ex1 {
  background-color: black;
  width: 900px;
  height: 500px;
  overflow: scroll;
  margin: 50px;
}
</style>

<body style="background-color: #565c60; font-family: Georgia, 'Times New Roman', Times, serif; color: white; margin:0"></body>
    <div class="ex1">
        {% if output %}<h3>{{ output }}</h3>{% endif %}
        {% if exit_code %}<h3> The command returned an error: {{ error_msg }}</h3>{% endif %}
    </div>
    <div class="container">
        <a class="button button--wide button--white" href="home.html" title="Home" style="color: white; margin: 60px;">
            <span class="button__inner">
          Home
        </span>
        </a>
    </div>
</body>
</html>
RobTheRobot16
  • 323
  • 4
  • 24
  • 3
    Yes, it's possible. I would recommend using Server Side Events to do this, as you only require one way communication and websockets would introduce a new protocol and be overkill. Here is an answer of mine which overviews making SSEs in its simplest form https://stackoverflow.com/a/62077516/5180047 – Nick Brady Jul 24 '20 at 14:19
  • @NickBrady that's really interesting ! Do I need an ASGI server, or will a plain WSGI server fit the bill ? – Mario Orlandi Jul 24 '20 at 16:52
  • It would also be possible with REST APIs, requesting updates about the status of the command from the HTTP server every few seconds. Simple, but I'm not sure if such a thing would be practical. – idontknow Jul 24 '20 at 17:29
  • as the previous user said, rest APIs would work too if you're okay with polling from the client, and is the simplest route. You just have to live with the event not coming _right_ as they occur. That's a great question about WSGI vs ASGI. It'll run on WSGI but a `while` loop will block a WSGI process I think, so you might want to run it as a thread on your process I'd do some testing there and try doing some load testing and some more research based on your needs. I haven't implemented SSE on a large scale, just for a small app where the request load was low so I didn't need to worry much. – Nick Brady Jul 24 '20 at 19:53
  • @NickBrady - Do you have a code example for Django? I'm trying to understand how I can implement what I've already got above into a working solution that simply streams the console output on a HTML page. After reading the suggested answer below, websockets seem a) overkill, and b) overly complicated to implement for what I'm trying to achieve. It's also annoying that every tutorial on the Internet points to a chat window and literally nothing is explained whatsoever :( For a complete novice it's an absolute nightmare. – RobTheRobot16 Jul 30 '20 at 15:39
  • 1
    I do not, sorry. SEE uses the HTTP protocol, so all you really need to do is send a generic text/stream response. I wouldn't think too much about Django or not Django. Research how to return a text stream, and then do it in a similar way to my example. Don't forget about running it on a thread :) – Nick Brady Jul 30 '20 at 18:41
  • Thanks, @NickBrady! You've definitely pointed me in the right direction, so I'll do some research :) – RobTheRobot16 Jul 31 '20 at 08:50
  • 1
    @RobTheRobot16 glad to hear it! Make sure to post your solution here as an answer ;) seems to be a popular question. Feel free to contact me personally (can find a simple little site I have in my bio) if needed. – Nick Brady Jul 31 '20 at 13:43

2 Answers2

5

You could simplify your task using StreamingHttpResponse and Popen:

def test_iterator():
    from subprocess import Popen, PIPE, CalledProcessError

    with Popen(['ping', 'localhost'], stdout=PIPE, bufsize=1, universal_newlines=True) as p:
        for line in p.stdout:
            yield(line + '<br>') # process line here

    if p.returncode != 0:
        raise CalledProcessError(p.returncode, p.args)

def busy_view(request):
    from django.http import StreamingHttpResponse
    return StreamingHttpResponse(test_iterator())

StreamingHttpResponse expects an iterator as its parameter. An iterator function is one which has a yield expression (or a generator expression), and its return value is a generator object (an iterator).

In this example, I simply echo the ping command to prove it works.

Substitute ['ping', 'localhost'] by a list (it has to be a list if you pass parameters to the command - in this case, localhost). Your original ['python', script] should work.

If you want to know more about generators, I would recommend Trey Hunner's talk, and also strongly that you read chapter 14 of Fluent Python book. Both are amazing sources.

Disclaimer:

Performance considerations

Django is designed for short-lived requests. Streaming responses will tie a worker process for the entire duration of the response. This may result in poor performance.

Generally speaking, you should perform expensive tasks outside of the request-response cycle, rather than resorting to a streamed response.

Niloct
  • 9,491
  • 3
  • 44
  • 57
4

What you would want is websockets, or Channels as they're known in Django.

https://channels.readthedocs.io/en/latest/

This allows you to send messages from the backend to the frontend without having to pull the messages on the frontend or reload the page.

Something worth mention is that you could also stream the output to multiple clients and also send back commands to your backend.

Approach tailored to your code

Please notice, this is untested as I do not have access to your code and therefor you might need some minor adjustments, I believe however the provided code should illustrate the concept.

Settings.py

INSTALLED_APPS = (
#Other installed Apps
       'Channels',
)
CHANNEL_LAYERS = {
      "default": {
          "BACKEND": "asgiref.inmemory.ChannelLayer",
            "ROUTING": "django_channels.routing.channel_routing",
      },
}

routing.py (add file in same folder as settings.py)

from django_channels_app.consumers import message_ws, listener_add, listener_discconect

channel_routing = [
      route("websocket.receive", message_ws),
      route("websocket.disconnect", listener_discconect),
      route("websocket.connect", listener_add),
]

In your module:

import threading
from channels import Group

class PreserializeThread(threading.Thread):
    def __init__(self, request, *args, **kwargs):
        self.request = request
        super(PreserializeThread, self).__init__(*args, **kwargs)

    def run(self):
        GenerateProjectConfig(request)
        home = os.getcwd()
        project_id = request.session['projectname']
        staging_folder = home + "/staging/" + project_id + "/"
        output = ""
        os.chdir(staging_folder)
        script = home + '/webscripts/terraformdeploy.py'
        try:
            output = subprocess.check_output(['python', script], shell=True)
            Group("django_channels_group").send({
                "text": output,
            })

            # NOTICE THIS WILL BLOCK; 
            # You could try the following, untested snippet


#    proc = subprocess.Popen(['python', script], shell=True, #stdout=subprocess.PIPE)
#    
#    line = proc.stdout.readline()
#    while line:
#        line = proc.stdout.readline()
#        Group("django_channels_group").send({
#                        "text": line,
#                    })
#    Group("django_channels_group").send({
#        "text": "Finished",
#    })
        except subprocess.CalledProcessError:
            exit_code, error_msg = (
                output.returncode,output.output)
        os.chdir(home)

def listener_add(message):
    Group("django_channels_group").add(
        message.reply_channel)

def listener_discconect(message):
    Group("django_channels_group").discard(
        message.reply_channel)

def message_ws(message):
    Group("django_channels_group").send({
          "text": "My group message",
     })

def projectprogress(request):
    ProgressThread(request).start()
    return render(request, 'projectprogress.html', locals())

html

<style>
  div.ex1 {
  background-color: black;
  width: 900px;
  height: 500px;
  overflow: scroll;
  margin: 50px;
}
</style>

<body style="background-color: #565c60; font-family: Georgia, 'Times New Roman', Times, serif; color: white; margin:0"></body>
    <div id="output">
        
    </div>
    <div class="container">
        <a class="button button--wide button--white" href="home.html" title="Home" style="color: white; margin: 60px;">
            <span class="button__inner">
          Home
        </span>
        </a>
    </div>
</body>
</html>

<script>
socket = new WebSocket("ws://127.0.0.1:8000/"); #Or your server IP address
socket.onmessage = function(e) {
    const data = JSON.parse(e.data);
    document.querySelector('#ouput').value += (data.message + '\n');
}
socket.onopen = function() {
    socket.send("Test message");
}
</script>

More generic answer

Backend:

chat/consumers.py:

import json
from asgiref.sync import async_to_sync
from channels.generic.websocket import WebsocketConsumer

class ChatConsumer(WebsocketConsumer):
    def connect(self):
        self.room_name = self.scope['url_route']['kwargs']['room_name']
        self.room_group_name = 'chat_%s' % self.room_name

        # Join room group
        async_to_sync(self.channel_layer.group_add)(
            self.room_group_name,
            self.channel_name
        )

        self.accept()

    def disconnect(self, close_code):
        # Leave room group
        async_to_sync(self.channel_layer.group_discard)(
            self.room_group_name,
            selfreturn render(request, 'projectprogress.html', locals()).channel_name
        )

    def send_message(self, event):
        message = event['message']

        # Send message to WebSocket
        self.send(text_data=json.dumps({
            'message': message
        }))

mysite/settings.py:

# Channels
ASGI_APPLICATION = 'mysite.routing.application'
CHANNEL_LAYERS = {
    'default': {
        'BACKEND': 'channels_redis.core.RedisChannelLayer',
        'CONFIG': {
            "hosts": [('127.0.0.1', 6379)],
        },
    },
}

mysite/routing.py:

from channels.auth import AuthMiddlewareStack
from channels.routing import ProtocolTypeRouter, URLRouter
import chat.routing

application = ProtocolTypeRouter({
    # (http->django views is added by default)
    'websocket': AuthMiddlewareStack(
        URLRouter(
            chat.routing.websocket_urlpatterns
        )
    ),
})

chat/routing.py:

from django.urls import re_path

from . import consumers

websocket_urlpatterns = [
    re_path(r'ws/chat/(?P<room_name>\w+)/$', consumers.ChatConsumer),
]

Frontend:

<script>
const chatSocket = new WebSocket(
            'ws://'
            + window.location.host
            + '/ws/chat/'
            + roomName
            + '/'
        );

        chatSocket.onmessage = function(e) {
            const data = JSON.parse(e.data);
            document.querySelector('#chat-log').value += (data.message + '\n');
        };
</script>
alexisdevarennes
  • 5,437
  • 4
  • 24
  • 38
  • Thanks; I'll have to do some research with this considering I'm completely new to Django and the usage of Channels. Do you know where I ought to be putting the subprocess command? Within the consumers.py file somewhere? – RobTheRobot16 Jul 30 '20 at 15:10
  • 1
    @RobTheRobot16 sorry for not replying to your comment earlier, I'll update my answer as how to do that. – alexisdevarennes Jul 31 '20 at 12:19
  • 1
    @RobTheRobot16 I've updated my answer, you might want to check once more if you had already seen the update as I corrected some typos. – alexisdevarennes Jul 31 '20 at 12:52
  • 1
    Added a comment regarding making it stream the output instead of waiting for it to finish. – alexisdevarennes Jul 31 '20 at 12:58
  • Thanks, @alexisdevarennes! I've had to update my settings to include "ASGI_APPLICATION = 'django_forms.routing.application'" (as per https://stackoverflow.com/questions/58841118/djangorediscommanderror-you-have-not-set-asgi-application-which-is-needed-to), but I'm a bit confused now as to where ProtocolTypeRouter comes into play in the routing.py file of your specific example. In the generic answer you make use of URL patterns, but in the more specific answer you simply list out the routes. Could you explain the difference, please? – RobTheRobot16 Aug 04 '20 at 10:46