1

I have to read data from log files present in realtime. I used the code mentioned in Reading multiple files in real time? however I am able to read only the first file i.e system.log. How to read all the files iteratively i.e first system.log and then wifi.log and repeat the same process again.

import time
import glob
import threading

def follow(thefile):
   thefile.seek(0,2)
    while True:
        line = thefile.readline()
        if not line:
            time.sleep(0.1)
            continue
        yield line


 lock = threading.Lock()

 def printFile(logfile):
    loglines = follow(logfile)
    for line in loglines:
        lock.acquire()
        print (line)
        lock.release()



  if __name__ == '__main__':
      files=['system.log','wifi.log']
      for log in files:
          logfile = open(log, "r")
          t = threading.Thread(target = printFile,args = (logfile,))
          t.start()
Seeker
  • 33
  • 4
  • If you are on unix you can use system calls to achieve this very easily. Here is an example: https://stackoverflow.com/questions/12523044/how-can-i-tail-a-log-file-in-python – mama Aug 13 '21 at 17:25
  • @mama I have to use this code for linux systems – Seeker Aug 13 '21 at 17:27
  • linux is unix: so you can use the link from my first comment. – mama Aug 13 '21 at 17:28
  • I am able to do that for single file. I have to do the same for multiple files. – Seeker Aug 13 '21 at 17:32
  • As you see on the question one of the methods are `Non-Blocking` so you can use it to follow multiple files concurrently. I can make an example for you in a moment. – mama Aug 13 '21 at 17:33
  • @mama Can you share the code for the same. – Seeker Aug 13 '21 at 18:00
  • After looking a bit on the answers on the link i showed you I saw an answer lower down which seemed much better. I optimized it a bit for your usecase., – mama Aug 13 '21 at 18:30

1 Answers1

2

You can use the asyncio library to create a concurrent function to follow the file's tail and then print it.

import asyncio

async def follow(thefile):
    thefile.seek(0,2)
    while True:
        line = thefile.readline()
        if not line:
            await asyncio.sleep(0.1)
            continue
        yield line.strip()

async def main(path):
    async for x in follow(open(path)):
        print(x)

for path in ['hej', 'med', 'dig']:
    asyncio.ensure_future(main(path))

loop = asyncio.get_event_loop()
loop.run_forever()

You could also skip the main function and the yield in the follow function, and just print the output directly in the follow function instead of passing it.

mama
  • 2,046
  • 1
  • 7
  • 24