2

I have an application which logs data to a log file (data.log). This file is capped to 2 MB. After it reaches the cap, it is renamed to a backup file (data.log.bak) and creates a new log file with the same name (data.log) and starts logging there. If the cap is reached for the new file, it again saves it to data.log.bak and the old log file gets deleted.

I need to write a python2.7 script which is looking for a particular string in the log. It should execute in parallel with the main application to account for the deleted bak files. The script should work on PC and Mac.

How do I deal with reading a file that gets renamed/deleted. I used this code a bit: https://stackoverflow.com/a/39213830/769078

def open_file(self):
    if sys.platform == 'win32':
        # get an handle using win32 API, specifying the SHARED access!
        handle = win32file.CreateFile(self.filename,
                                      win32file.GENERIC_READ,
                                      win32file.FILE_SHARE_DELETE |
                                      win32file.FILE_SHARE_READ |
                                      win32file.FILE_SHARE_WRITE,
                                      None,
                                      win32file.OPEN_EXISTING,
                                      0,
                                      None)
        # detach the handle
        detached_handle = handle.Detach()
        # get a file descriptor associated to the handle\
        file_descriptor = msvcrt.open_osfhandle(detached_handle, os.O_RDONLY)
        # open the file descriptor
        f = os.fdopen(file_descriptor)
    else:
        f = open(self.filename, 'r')

    return f


def follow(self):
    if not os.path.isfile(self.filename):
        return False

    current = self.open_file()
    try:
        ino = os.fstat(current.fileno()).st_ino
        while True:
            line = current.readline().strip()
            where = current.tell()
            print '[log]', line
            if line:  # Read a line
                if 'error' in line:  # If error found in line
                    self.found = True
                    print '[Debug] Found exception'
                    break
            elif self.stop:  # no new line and no new file. Stop reading if not stop
                print '[Debug] Stopping'
                break
            elif os.stat(self.filename).st_ino != ino:  # Reached end of file. Check if new file exists
                print '[Debug] Detected new file'
                new = self.open_file()
                current.close()
                current = new
                ino = os.fstat(current.fileno()).st_ino
            else:  # no new line and no new file and not stop. Sleep for a second
                print '[Debug] sleeping'
                current.seek(where)
                time.sleep(2)
    finally:
        # close the file
        current.close()

I use thread to run the script in parallel to the main application. I skipped that code here.

* UPDATE *

Using ino didn't work for me. I might have made a mistake. Instead, I relied on modified time.

def follow(self):
    if not os.path.isfile(self.filename):
        return False

    current = self.open_file()
    bak_modified_time = None
    try:
        while True:
            line = current.readline().strip()
            where = current.tell()
            if line:  # Read a line
                if 'error' in line:  # If error found in line
                    self.found = True
                    print '[Debug] Found exception'
                    break
            elif self.stop:  # no new line and no new file. Stop reading if not stop
                print '[Debug] Stopping'
                break
            elif ((not bak_modified_time and os.path.exists(self.bak_filename)) or
                  (bak_modified_time and os.path.exists(self.bak_filename) and
                   os.stat(self.bak_filename).st_mtime != bak_modified_time))
                print '[Debug] Detected new file'
                new = self.open_file()
                current.close()
                current = new
                bak_modified_time = os.stat(self.bak_filename).st_mtime
            else:  # no new line and no new file and not stop. Sleep for a second
                print '[Debug] sleeping'
                current.seek(where)
                time.sleep(2)
    finally:
        # close the file
        current.close()
Sri
  • 405
  • 6
  • 16
  • You might want to try copytruncate method of rolling log files. Then the only change you need is to open the log file in append mode. – Hitobat May 11 '18 at 20:52
  • At the time of backing up, the main application looks for data.log.bak. If it exists, it deletes it. Then, rename data.log to data.log.bak and then create a new data.log file. The only thing I can change is the script. I cannot edit the main application. – Sri May 11 '18 at 21:30

0 Answers0