0

I am working on a python script, where I will be passing a directory, and I need to get all log-files from it. Currently, I have a small script which watches for any changes to these files and then processes that information.

It's working good, but it's just for a single file, and hardcoded file value. How can I pass a directory to it, and still watch all the files. My confusion is since I am working on these files in a while loop which should always stay running, how can I do that for n number of files inside a directory?

Current code :

import time

f = open('/var/log/nginx/access.log', 'r')
while True:
    line = ''
    while len(line) == 0 or line[-1] != '\n':
        tail = f.readline()
        if tail == '':
            time.sleep(0.1)          # avoid busy waiting
            continue
        line += tail
        print(line)
        _process_line(line)

Question was already tagged for duplicate, but the requirement is to get changes line by line from all files inside directory. Other questions cover single file, which is already working.

We are Borg
  • 5,117
  • 17
  • 102
  • 225
  • Have you considered using [inotify](https://man7.org/linux/man-pages/man7/inotify.7.html) which is designed for just this usecase? – SiHa Nov 12 '20 at 07:52
  • Does this answer your question? [Monitoring contents of files/directories?](https://stackoverflow.com/questions/597903/monitoring-contents-of-files-directories) – SiHa Nov 12 '20 at 07:56
  • @SiHa No, any answer on that page are specifically for single files. How do I utilize inotify to get changes line by line as I mentioned in the code? – We are Borg Nov 12 '20 at 08:03
  • Doing it line-by-line for a directory full of files is an incredibly inefficient way to do it, and there are already mechanisms available to do this for you without having to re-invent the wheel. – SiHa Nov 12 '20 at 08:15
  • @SiHa : I need the changes in file. See this : https://stackoverflow.com/questions/25350252/any-ways-to-show-file-changes-with-pyinotify-for-example Inotify doesn't give the changes in file. – We are Borg Nov 12 '20 at 08:29

2 Answers2

0

Try this library: watchdog.

Python API library and shell utilities to monitor file system events.

https://pythonhosted.org/watchdog/

Simple example:

import sys
import time
import logging
from watchdog.observers import Observer
from watchdog.events import LoggingEventHandler

if __name__ == "__main__":
    logging.basicConfig(level=logging.INFO,
                        format='%(asctime)s - %(message)s',
                        datefmt='%Y-%m-%d %H:%M:%S')
    path = sys.argv[1] if len(sys.argv) > 1 else '.'
    event_handler = LoggingEventHandler()
    observer = Observer()
    observer.schedule(event_handler, path, recursive=True)
    observer.start()
    try:
        while True:
            time.sleep(1)
    except KeyboardInterrupt:
        observer.stop()
    observer.join()
zvi
  • 3,677
  • 2
  • 30
  • 48
  • I think the path u mentioned is the directory which I should pass. Where do I get the changes happening inside the file? – We are Borg Nov 12 '20 at 07:49
  • https://medium.com/@xiaoouwang/create-a-watchdog-in-python-to-look-for-filesystem-changes-aaabefd14de4 – zvi Nov 12 '20 at 08:33
  • This doesn't give the actual changes done... Whether the file was modified or not is not relavant, only the new changes. :-) – We are Borg Nov 12 '20 at 08:41
  • you can save the last line you read, and on the event - read from there – zvi Nov 12 '20 at 08:47
0

I'm not completely sure if I understand what your trying to do, but maybe use:

while True: 
    files = os.listdir(directory) 
    for file in files:
 
    --You're code for checking contents of the file--
JerAnsel
  • 13
  • 4