Reading in a file being constantly updated in python

Question

I am trying to read in a log file in python that is updated every 20 seconds. So far I am only able to read a static file. I am trying to read in these values to plot them on a graph using HighCharts. I am able to plot the values using a static file. However I would like the graph to be real time. Could anyone help me with this?

with open(self.filename) as f:
        cpu = []
        idle = []
        wait = []
        ld5 = []
        ld10 = []
        ld15 = []
        load = []
        swapu = []
        swapf = []

        #need to remove all the white space in the file
        for line in f:
            if line is not None:
                    stripped = line.strip()
                    dstat_tokens = self.whitespace_re.split(stripped)           
            #dstat_tokens = self.whitespace_re.split(stripped)
                    dstat_tokens = line.split(',')
                    (timestamp) = dstat_tokens[0:1]
                    (system,user,idl, wt, hwi, sfi) = dstat_tokens[1:7];
                    (diskread,diskwrite) = dstat_tokens[7:9]
                    (pagein,pageout) = dstat_tokens[9:11]
                    (load5,load10,load15) = dstat_tokens[11:14]
                    (used,buffers,cache,free) = dstat_tokens[14:18]
                    (swapused,swapfree) = dstat_tokens[18:20]
                    (interrupts,contextswitches) = dstat_tokens[20:22]
                    (receive,send) = dstat_tokens[22:24]
                    (runnable,uninterruptable,new) = dstat_tokens[24:27]

                #print "Stripping %s" % timestamp[0]
                    ts = time.strptime(timestamp[0], "%d %b %Y %H:%M:%S CET")
                    dt = datetime.fromtimestamp(time.mktime(ts))
                    returned_time = dt.strftime("%d/%m/%Y %H:%M:%S")
                #print returned_time

                    cpu.append({"ts": returned_time, "value": float(system)})
                    idle.append({"ts": returned_time, "value": float(idl)})
                    wait.append({"ts": returned_time, "value": float(wt)})
                    ld5.append({"ts": returned_time, "value": float(load5)})
                    ld10.append({"ts": returned_time, "value": float(load10)})
                    ld15.append({"ts": returned_time, "value": float(load15)})
                    load.append({"ts": returned_time, "value": float(load5)})
                    swapu.append({"ts": returned_time, "value": float(swapused)})
                    swapf.append({"ts": returned_time, "value": float(swapfree)})

            self.data["cpu"] = cpu
            self.data["idle"] = idle
            self.data["load5"] = ld5
            self.data["load10"] = ld10
            self.data["load15"] = ld15
            self.data["swapused"] = swapu
            self.data["swapfree"] = swapf`

Answer 1

The way I understand your question is that you want to update your graph every time your log file is updated. You could do that by storing the length of your file in a txt document every time you read and then when you update your graph every x seconds you check that txt document and get the file position from it and seek() to that position.

f = open(log, 'r')
f2 = open(log_file_position,'r+')
last_position = f2.readline()
f.seek(last_position)
#do stuff to log file so you will have read to the end
f2.truncate(0)
f2.write(f.tell()) #f.tell() gives you the current position in bytes from the beginning of the file
f2.close()

I'm rusty on my python and can't remember how to do this exactly but that's the general idea. If someone knows how to actually do this then please update my code. Let me know if you have any questions and I'll try to clarify.