Output log file through Ajax in Django

Question

I have followed an SO's accepted answer on how to read the log file in Django from /var/log/gateway from here and I managed to output the file on terminal like here.

2013-05-09T11:15:02.539091+08:00 localhost gateway[5205]: System starting up...
2013-05-09T12:57:44.160246+08:00 localhost gateway[5205]: System start complete.
2013-05-09T15:13:47.428553+08:00 localhost gateway[4777]: * Unable to connect to device /home/smartsensor1. Device may be offline. *

The next step is, I want to output the log file and display it in html, I did it with slight modification from the original code like this.

def Logs(request):
    with open('../../../../../var/log/gateway') as f:
        while True:
            line = f.readline()
            if line:
                print line
                return HttpResponse(line)

So on the client side, I put Ajax like this, based on another SO's accepted answer here.

$.ajax({
    type: "GET", 
    url : "{% url WebServiceApp.logging.Logs %}",
    success: function (data) {
            $("#output").append(data);
            setTimeout("doUpdate()", 2000);
    }
});
}

setTimeout("doUpdate()", 2000);

With this, the output data from Ajax kept on displaying the first line of the log file. Where in this case, is like this

2013-05-09T11:15:02.539091+08:00 localhost gateway[5205]: System starting up...
2013-05-09T11:15:02.539091+08:00 localhost gateway[5205]: System starting up...
2013-05-09T11:15:02.539091+08:00 localhost gateway[5205]: System starting up...

I know this occur because everytime ajax went to the server, the server does what it needs to do and send back the output which is the first line of the log file and output through the HttpResponse and completed the cycle and it never got a chance to do another line because it is completed already. When another query is done, it does the same thing again, and again.

So the possible solution is client ask the server one time, and the server keep output the log file line by line and send it out to client. I am not sure if this is even possible, so here I am asking any expert on how to possibly achieve the result where I can output the log file line by line/

Answer 1

Well, you can use an idea like this:

Using GET parameters you can control the view's method like this:

def Logs(request):
    whole_file = True
    if request.GET.get('whole_file') == 0:
        whole_file = False
    return_string = ''
    with open('../../../../../var/log/gateway') as f:
        while True:
            line = f.readline()
            if line:
                return_string += line + '<br>' # <br> is for breakline html, do it your way
                if not whole_file: break # break if you only need first line
    return HttpResponse(line)

This way, you include a variable inside the get parameter that tells your view whether to return the whole log file (first time) or just the first line (next times).

Then in your ajax you will have to include a way to insert GET parameters to the url. You use the {{url}} tag an I'm not quite sure how to do it with it but I'm sure you'll find something in the doc, if you can't then try some other approach like an static string (not very pretty) or create your own custom tag for it.

This would be an example of your possible javascript:

$.ajax({
    type: "GET", 
    url : "/ajax_log/?whole_file=0", //this will request the whole file
    success: function (data) {
            $("#output").append(data);
            setTimeout("doUpdate()", 2000);
    }
});
}

setTimeout("doNextUpdate()", 2000);

function doNextUpdate(){

    $.ajax({
        type: "GET", 
        url : "/ajax_log/?whole_file=1", //this will request the just the first line            success: function (data) {
                $("#output").append(data);
                setTimeout("doUpdate()", 2000);
        }
    });
    }
}

That is my idea to accomplish what you want. I hope it helps.

Answer 2

I've found a way to solve this is by doing the Server-Sent Event

On server side, I use django-sse that is dependent on sse

import os
from django.conf import settings
from django.views.generic import View
from django_sse.views import BaseSseView
import time
from django.utils.timezone import now
from datetime import datetime
from django.utils.timezone import now
from dateutil import parser

class MySseEvents(BaseSseView):
    def iterator(self):
        while True:
            if not os.path.exists('/var/log/gateway'):
                file('/var/log/gateway', 'w+').close()
            with open('/var/log/gateway') as f:
                while True:
                    #get the current time
                    currenttime = now()
                    #read a line from gateway
                    line = f.readline()
                    if line:
                        #split the line read into 4 section
                        str = line.split(' ',3)
                        #compare if the time from the log file is less than the current time
                        logtime = parser.parse(str[0])
                        if logtime >= currenttime:
                            #print when the log is written after the current time
                            output = '%s: %s' % (datetime.strftime(logtime,'%d/%m %H:%M %p'), str[3])
                            self.sse.add_message("line", output)
                            time.sleep(1)
                            yield

The if logtime >= currenttime: check that I did above is to make sure that html only print out the logs after the time the page is loaded not the whole log file.

Then on the HTML5 side I did similar to what the example made by the author shown on the first link that I have provided.

<div id="tab9" class="tab_content">
    <table>
        <tr>
            <td>
                <p> Shows the records of the logs on gateway.</p>
                <div style="border:1px solid; height: 200px; width: 850px; overflow: auto;" id="output"></div>
            </td>
        </tr>
    </table>
</div>
<script>
$(document).ready(function() {
    var source = new EventSource('/gtwy/logging/');
    var events_dom = $("#output");

    source.addEventListener("line", function(e) {
        events_dom.append(e.data + "<br>");
    });
});
</script>

Using this, I managed to display the log as it is available. The only problem that I face is to create the gateway file itself that it needs permission because I'm trying to create it in /var/log.