Check if a file has been written in last 5 minutes, to restart if the script is stuck

Question

I've an infinite while loop which fetches some data and writes it to a csv. Sometimes due to random reason, most likely because of memory shortage, the script gets stuck. So I've tried to restart the script if such a thing happens, with help of check file being written and this restart function which is used verbatim, I've arrived at the following code

from urllib2 import Request, urlopen
import pandas as pd

from datetime import datetime, timedelta
import time
import json
import os.path

import os
import sys
import psutil
import logging

currtime = time.localtime()
dct = time.strftime('%d%b%Y', currtime).upper()
datafile = 'Idx' + dct + '.csv'
csvfile = 'Index/' + datafile

ptime = datetime.now()
idx = pd.DataFrame()

last_size = os.stat(csvfile).st_size
last_time = ptime + timedelta(0,5*60)

while True:
    if (ptime.hour*60*60 + ptime.minute*60 + ptime.second) > stop:
        break
    else:
        try:
            ptime = datetime.now()
            idx = getIndexData()
            idx.to_csv(csvfile, index=False, mode='a', header=False)


            if not nosleep:
                end = datetime.now()
                tdiff = end-ptime
                time.sleep(scrapeGap - (tdiff.seconds + tdiff.microseconds*1.0/10**6))

        except Exception as e:
            with open("idx.log", "a") as lf:
                lf.write("Error at {0} \t {1}\n".format(str(ptime), str(e)))
            continue

        if ptime > last_time:
            if os.stat(csvfile).st_size - last_size > 0:
                continue
            else:
                restart_program()

            last_size = os.stat(csvfile).st_size
            last_time = ptime + timedelta(0,5*60)

Is this a correct way to construct the whole thing? I had no time test the restarting of the script, as I can't definitely trigger the script to get stuck.

Is pointers about any way to get information about what is causing the script to get stuck is also would be great help.

EDIT:

As @iAmTryingOK asked here is the csv file being written

OI.call,changeOI.call,volume.call,IV.call,LTP.call,netChange.call,bidQ.call,bidP.call,askP.call,askQ.call,strike,bidQ.put,bidP.put,askP.put,askQ.put,netChange.put,LTP.put,IV.put,volume.put,changeOI.put,OI.put,timestamp
18300.0,-450.0,8.0,0.0,2474.85,43.1,75.0,2459.6,2471.6,600.0,9300.0,150.0,3.35,3.5,675.0,0.15,3.5,36.06,139.0,-6300.0,56625.0,2019-05-01 09:00:29.624
0.0,0.0,0.0,0.0,0.0,0.0,375.0,2261.9,2695.95,300.0,9350.0,300.0,0.1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,2019-05-01 09:00:29.624
1125.0,0.0,0.0,0.0,2384.95,-0.55,75.0,2358.1,2383.65,75.0,9400.0,75.0,3.5,5.4,75.0,0.05,4.2,35.47,2.0,75.0,600.0,2019-05-01 09:00:29.624
0.0,0.0,0.0,0.0,0.0,0.0,375.0,2141.7,2555.9,4950.0,9450.0,150.0,0.55,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,2019-05-01 09:00:29.624
870225.0,4275.0,653.0,0.0,2264.95,-18.95,300.0,2264.1,2271.95,375.0,9500.0,2175.0,3.95,4.05,450.0,-0.55,3.95,33.75,1131.0,29100.0,293550.0,2019-05-01 09:00:29.624

would you not like us to help with the process becoming unresponsive rather than creating a workaround to allow things to fail? — iAmTryingOK, May 16 '19 at 06:52
@iAmTryingOK Sorry I'm unable to comprehend your comment. Can you please elaborate? — Frash, May 16 '19 at 10:42
I meant to ask, if the desired solution to your problem is preparing a mechanism to work when the file reading gets stuck as compared to enhancing the code to make sure that we only read the file when it's available. — iAmTryingOK, May 16 '19 at 10:44
@iAmTryingOK Well it's that file writing which gets stuck and primarily would like to restart when it gets stuck. Getting to know and fix the reason for it getting stuck would be better, but for now I'll do with just restarting without fixing the underlying cause. — Frash, May 17 '19 at 04:55
all right, can you share how the csv looks, maybe add a snippet of it — iAmTryingOK, May 17 '19 at 06:26

score 0 · Accepted Answer · answered Nov 30 '19 at 11:19

The bug was that when scrapeGap - (tdiff.seconds + tdiff.microseconds*1.0/10**6) resulted in a negative number it was putting off the execution for a negative count of time, which means it never break out of the time.sleep. This resulted in the code getting stuck. Putting a check before the time.sleep to sleep only for a positive number resolved the issue.

Buggy part:

time.sleep(scrapeGap - (tdiff.seconds + tdiff.microseconds*1.0/10**6))

Corrected part:

tslip = scrapeGap - (tdiff.seconds + tdiff.microseconds*1.0/10**6)
if tslip > 0:
   time.sleep(tslip)

The above edited code fragment solves the issue, adding this for future refernce.

Check if a file has been written in last 5 minutes, to restart if the script is stuck

1 Answers1