Monitoring if a file stopped writing in python

Question

I have a program that keeps writing to a file every second. File writing is happening in a thread parallel to the UI. Due to some hardware issue it stops writing sometimes of a day. I wanted to check if the file stopped writing, restart the program if it is not getting updated. I wanted to check the file's timestamp and see if it is not getting updated(and did not want to get to watchdog etc. coz I just needed if a file stopped writing.)

try:
    if time.time()>(os.stat(filename).st_mtime+2):
        raise ValueError("Yikes! Spike")
except ValueError:
    with open('errors.log','a') as log:
        log.write('Spike occured at '+ time.strftime(
        "%H:%M:%S")+' on '+datetime.date.today().strftime('%d/%m/%Y')+'\n')
        log.close()
    restart_program()

This block runs every second. But this backfired and when the app closes for restarting it keeps closing every second and doesn't start again. I get the exception message logged every second. I tried increasing the time difference but that didn't help.

Next I tried

ftimestamp = os.stat(filename).st_mtime
try:
    if os.stat(filename).st_mtime>=ftimestamp:
        ftimestamp = time.time()
        print "ftimestamp updated and all is well"
    else:
        ftimestamp = os.stat(filename).st_mtime
        raise ValueError("Yikes! Spike!")
        print "file time is behind"
except ValueError:
    with open('errors.log','a') as log:
        log.write('Spike occured at '+ time.strftime(
        "%H:%M:%S")+' on '+datetime.date.today().strftime('%d/%m/%Y')+'\n')
        log.close()
    restart_program()

I tried updating the variable "ftimestamp" to current time "time.time()" because the next comparison happens only after one second and I want the file time to be higher than the previous time comparison. (The block runs every second through wx.CallLater function).

My program fails still... and I am not able to understand where I am going wrong... Someone please help! Or is there a way of simply checking if the file stopped writing?

Answer 1

We can try checking for a change in file size as a possible solution by doing the following:

import os
from time import sleep
# other imports

while True:
    file1 = os.stat('file.txt') # initial file size
    file1_size = file1.st_size
 
    # your script here that collects and writes data (increase file size)
    sleep(1)
    file2 = os.stat('file.txt') # updated file size
    file2_size = file2.st_size
    comp = file2_size - file1_size # compares sizes
    if comp == 0:
        restart_program()
    else:
        sleep(5)

You may need to adjust the sleep() function accordingly these are just estimates that I'm using since I can't test your actual code. In the end this is infinite loop that will keep running as long as you want the script to keep writing.

Another solution is to update your code to:

import os
import sys
from time import sleep
# other imports

while True:
    file1 = os.stat('file.txt') # initial file size
    file1_size = file1.st_size
 
    # your script here that collects and writes data (increase file size)
    sleep(1)
    file2 = os.stat('file.txt') # updated file size
    file2_size = file2.st_size
    comp = file2_size - file1_size # compares sizes
    if comp == 0:
        sys.exit
    else:
        sleep(5)

Then use a secondary program to run your script as such:

import os
from time import sleep, strftime

while True:
    print(strftime("%H:%M:%S"), "Starting"))
    system('main.py') # this is another infinite loop that will keep your script running
    print(strftime("%H:%M:%S"), "Crashed"))
    sleep(5)

Answer 2

To determine whether the file changes on time in a GUI program, you could use standard tools for your event loop, to run a function every interval seconds e.g., here's how to do it in tkinter:

#!/usr/bin/env python3
import logging
import os
import sys
import tkinter
from datetime import datetime
from time import monotonic as timer, localtime

path = sys.argv[1]
interval = 120 # check every 2 minutes

def check(last=[None]):
    mtime = os.path.getmtime(path) # or os.path.getsize(path)
    logging.debug("mtime %s", datetime.fromtimestamp(mtime))
    if last[0] == mtime: #NOTE: it is always False on the first run
        logging.error("file metadata hasn't been updated, exiting..")
        root.destroy() # exit GUI
    else: # schedule the next run
        last[0] = mtime
        root.after(round(1000 * (interval - timer() % interval)), check)


logging.basicConfig(level=logging.DEBUG,
                    filename=os.path.splitext(__file__)[0] + ".log",
                    format="%(asctime)-15s %(message)s", datefmt="%F %T")
root = tkinter.Tk()
root.withdraw() # hide GUI
root.after(round(1000 * (interval - timer() % interval)), check) # start on boundary
root.mainloop()

You could use supervisord, or systemd, or upstart, etc to respawn your script automatically.

See How to run a function periodically in python.

Answer 3

Finally, after tinkering around with timestamp based options, the following seemed to work for me.

try:
    if time.time()-os.stat(filename).st_mtime>6:
        touch(filename)
        raise ValueError("Yikes! Spike")
except ValueError:
    with open('errors.log','a') as log:
        log.write('Spike/App restarting occured at '+ time.strftime(
                "%H:%M:%S")+' on '+datetime.date.today().strftime('%d/%m/%Y')+'\n')
        log.close()
    restart_program()

Earlier, the problem was it would detect that the file stopped writing with the given time interval and continue to satisfy the same.

time.time()-os.stat(filename).st_mtime>6 

But once this condition is satisfied, unless the file timestamp is updated it continues to satisfy this condition and would keep restarting the program. Now in my solution, I 'touched' the file once (touch used from here) the condition is satisfied and now it works as expected.

Thank you all for your inputs.