Read Apache log file realtime and send an e-mail-Python

Question

Need to read an Apache log file realtime from the server and if some string is found an e-mail has to be sent. I have adopted the code found here to read the log file. Next how to send this e-mail. Do I have to issue a sleep command? Please advice.

Note: Since this is real time, after sending the e-mail python program has to begin reading the log file again. This process continues.

import time
import os

#open the file
filename = '/var/log/apache2/access.log'
file = open(filename,'r')

while 1:
    where = file.tell()
    line = file.readline()
    if not line:
        time.sleep(1)
        file.seek(where)
    else:
        if 'MyTerm' in line:
            print line

Answer 1

Well, if you want it real time and not to be stuck on sending mails you could start a separate thread to send the email. Here is how you use threads in python (thread and threading):

http://www.tutorialspoint.com/python/python_multithreading.htm

Next, you can easily send an email in python using the smtplib. Here is another example from the same website (which I use and it is pretty good):

http://www.tutorialspoint.com/python/python_sending_email.htm

Well, you need to do this to speed up as much as possible the log-reading thread and to be sure it won't wait for mailing.

Now some pitfalls you have to take care of:

1. You must be careful with starting too many threads. For instance you are parsing (let's just assume for the moment) the log every 1 second, but sending an email takes 10 seconds. It is easy to see that (this is an exaggerated example, of course) you will start many threads and you will fill the available resources. I don't know how many times the string you are expecting will pop out each second, but it is a scenario you must consider.

2. Again depending on the workload, you can implement a streaming algorithm and avoid emails entirely. I don't know if it applies in your case, but I prefer to remind you about this scenario too.

3. You can create a queue and put a certain number of messages in it and send them together, thus avoiding sending many mails at once (again assuming you don't need to trigger an alarm for every single occurrence of your target string you have).

UPDATE

If you really want to create the perfect program you can do something else by using event triggering when the log file is modified. This way, you will avoid sleep entirely and each time something has been added to the file, python will be called and you can parse the new content and send the email if required. Take a look at watchdog:

http://pythonhosted.org/watchdog/

and this: python pyinotify to monitor the specified suffix files in a dir

https://github.com/seb-m/pyinotify