Read new line with pynotify

Question

I'm trying to display a new line that was added to the file. So imagine that I have a sample_file.txt :

 1. line 1
 2. line 2
 3. line 3

I want to check if this file got a new line, and than display that line (without printing all the file again)

#!/usr/bin/python

import os
import pyinotify
from datetime import datetime
import time

PATH = os.path.join(os.path.expanduser('~/'), 'sample_file.txt')

m = pyinotify.WatchManager()
m.add_watch(PATH, pyinotify.ALL_EVENTS, rec=True)
notifier = pyinotify.Notifier(m, pyinotify.ProcessEvent(), 0,  0, 100)

f = open(PATH)
for line in f.readlines():
    print line

while True:
    time.sleep(5)
    try:
        if notifier.check_events():
            # RIGHT HERE:
            # HOW TO DO SOMETHING LIKE
            # f.last() ???
            print f.next()
        else:
            print 'waiting for a line....'
    except KeyboardInterrupt:
        notifier.stop()
        break

What I was thinking is to read all the lines somewhere before while loop, and than print the next line, but something wrong in my code, and it checks for f.next() right after it comes to the loop.

Answer 1

I'll address the two issues:

• how to implement tail on a file,
• and how to use the pyinotify module.

Tail on a file

In your code, you need to:

• try and read as much full lines as you can using read or readlines,
• rewind the file to the start of the last incomplete line until you can print it using seek.

This translates for example into:

f = open(PATH)
for line in f.readlines():
    print line[:-1]

while True:
    time.sleep(5)
    try:
        line_start = f.tell()
        new_lines = f.read()
        last_n = new_lines.rfind('\n')
        if last_n >= 0:
            # We got at least one full line
            line_start += last_n + 1
            print new_lines[:last_n]
        else:
            # No complete line yet
            print 'no line'
        f.seek(line_start)
    except KeyboardInterrupt:
        notifier.stop()
        break

You can find more examples here, although some do not address additions to the file which do not end with a newline:

• https://github.com/kasun/python-tail/blob/master/tail.py
• http://code.activestate.com/recipes/157035-tail-f-in-python/

And some alternatives here How can I tail a log file in Python?

Pyinotify loop

You should also move your code inside pyinotify's event handlers as explained in the documentation.

check_events returns True if there are events to be processed, but it does not actually process the events, so by itself it will always return True until events have been processed.

Also, try and avoid while/sleep loops. Inotify adds the capability to handle an event as soons as it is received, without compromising resources. A while/sleep loop will be less reactive.

Below are the two first methods form the Short Tutorial on pyinotify.

1. Monitoring endlessly

This is the preferred method if you have no other event loop, as it will be the most reactive:

PATH = os.path.join(os.path.expanduser('~/'), 'experiments', 'testfile')

class EventHandler(pyinotify.ProcessEvent):
    def __init__(self, *args, **kwargs):
        super(EventHandler, self).__init__(*args, **kwargs)
        self.file = open(PATH)
        self.position = 0
        self.print_lines()

    def process_IN_MODIFY(self, event):
        print 'event received'
        self.print_lines()

    def print_lines(self):
        new_lines = self.file.read()
        last_n = new_lines.rfind('\n')
        if last_n >= 0:
            self.position += last_n + 1
            print new_lines[:last_n]
        else:
            print 'no line'
        self.file.seek(self.position)

wm = pyinotify.WatchManager()
handler = EventHandler()
notifier = pyinotify.Notifier(wm, handler)
wm.add_watch(PATH, pyinotify.IN_MODIFY, rec=True)
notifier.loop()

2. Monitoring periodically

If you already have a processing loop, then it's just a matter of calling process_events periodically. The EventHandler class is the same as in method 1, but now instead of calling notifier.loop() we add a small timeout to the notifier, and implement our own event loop.

...

wm = pyinotify.WatchManager()
handler = EventHandler()
notifier = pyinotify.Notifier(wm, handler, timeout=10)
wm.add_watch(PATH, pyinotify.IN_MODIFY, rec=True)

while True:
    # Do something unrelated to pyinotify
    time.sleep(5)

    notifier.process_events()
    #loop in case more events appear while we are processing
    while notifier.check_events():
        notifier.read_events()
        notifier.process_events()