I'm trying to find a nice way to read a log file in real time using python. I'd like to process lines from a log file one at a time as it is written. Somehow I need to keep trying to read the file until it is created and then continue to process lines until I terminate the process.
Is there an appropriate way to do this?
Take a look at this PDF starting at page 38, ~slide I-77 and you'll find all the info you need. Of course the rest of the slides are amazing, too, but those specifically deal with your issue:
import time
def follow(thefile):
thefile.seek(0,2) # Go to the end of the file
while True:
line = thefile.readline()
if not line:
time.sleep(0.1) # Sleep briefly
continue
yield line
You could try with something like this:
import time
while 1:
where = file.tell()
line = file.readline()
if not line:
time.sleep(1)
file.seek(where)
else:
print line, # already has newline
Example was extracted from here.
As this is Python and logging tagged, there is another possibility to do this.
I assume this is based on a Python logger, logging.Handler based.
You can just create a class that gets the (named) logger instance and overwrite the emit function to put it onto a GUI (if you need console just add a console handler to the file handler)
Example:
import logging
class log_viewer(logging.Handler):
""" Class to redistribute python logging data """
# have a class member to store the existing logger
logger_instance = logging.getLogger("SomeNameOfYourExistingLogger")
def __init__(self, *args, **kwargs):
# Initialize the Handler
logging.Handler.__init__(self, *args)
# optional take format
# setFormatter function is derived from logging.Handler
for key, value in kwargs.items():
if "{}".format(key) == "format":
self.setFormatter(value)
# make the logger send data to this class
self.logger_instance.addHandler(self)
def emit(self, record):
""" Overload of logging.Handler method """
record = self.format(record)
# ---------------------------------------
# Now you can send it to a GUI or similar
# "Do work" starts here.
# ---------------------------------------
# just as an example what e.g. a console
# handler would do:
print(record)
I am currently using similar code to add a TkinterTreectrl.Multilistbox for viewing logger output at runtime.
Off-Side: The logger only gets data as soon as it is initialized, so if you want to have all your data available, you need to initialize it at the very beginning. (I know this is what is expected, but I think it is worth being mentioned.)
I stumbled upon this question cause I wanted to output the log line to the browser/webservice. The top answer got me most of the way there but my webservice would just not send the lines to the browser. What the issue was actually not thinking about async vs not-async. This took an entire afternoon to figure out, so I wanted to add my addition here.
First, we have to convert the follow() method into an asynchronous one, so use asyncio.sleep instead:
import asyncio
from io import TextIOWrapper
async def follow(thefile: TextIOWrapper):
thefile.seek(0,2) # Go to the end of the file
while True:
line = thefile.readline()
if not line:
await asyncio.sleep(0.1) # Sleep briefly
continue
yield line
follow() became an Asynchronous Generator, which means to use the result we'll have to use anext():
with open("text.txt", "r") as log_file:
print("opened file")
follow_generator = follow(log_file)
while True:
line = await anext(follow_generator)
# do whatever you want with line here
# here I am sending the new log line to my websocket
await websocket.send_text(str(line))
I made a demo which uses FastAPI.
Maybe you could do a system call to
tail -f
using os.system()
