Recommended method to pool messages in a open file loop for python

Question

I have a log monitoring python script I'm working on. It sends messages when entries with the text "disconnected" are found. The script reads the log until it comes across a "End of log file" message. In its current state it sends the messages as they come across. This isn't optimal and I need to pool them for say 5 minutes before sending the collected entries. I am not sure what the best method for doing this is. Here is a simplified version of what I am trying to do. So far I’ve tried time.sleep and an elaborate incremental counter to no avail.

# Open log
f = open(log, 'r')

# Start while loop and read line
while(1):

    # Check for Disconnected 
    if line.find("disconnected") != -1:

    ltime = time.time()
    print ("Disconnected Found in Log at " + ltime)

    # Check for end of log file
    if line.find("End of file") != -1:

    # End script
    break

you might find watchdog useful http://stackoverflow.com/questions/32923451/how-to-run-an-function-when-anything-changes-in-a-dir-with-python-watchdog/32923569#32923569 — Padraic Cunningham, Apr 11 '16 at 18:43

score 1 · Answer 1 · answered Apr 12 '16 at 13:55

Would this work for you?

Save current time in a variable and create an empty list before while loop.
Every time you find a "disconnect", append the message to the list.
In each iteration in the while loop, you compare the current time to the timestamp you saved (in step 1) - if more than 5 mins (300 sec) has passed you iterate through the list and send all messages.
Empty the list and save the current time in the variable from step 1.

Does this make sense? I can elaborate further if needed.

This is fairly close to what I ended up doing. – oljones Apr 12 '16 at 16:41 — oljones, Apr 12 '16 at 16:41

score 0 · Answer 2 · answered Apr 12 '16 at 16:47

This is what I ended up doing.

# Disconnects that have already been recorded
rec_disconnects = []

# Disconnects to msg
send_disconnects = []

# Start while loop
while(1):

    # Wait 5 minutes
    time.sleep(300)

    # If list isn't empty
    if send_disconnects:

        # Join all the disconnects from the send list
        msg = '\n'.join(send_disconnects)

        # print the disconnects
        print (msg)

        # Clear the list
        send_disconnects = []

    # Open the log file and read the lines
    with open(log) as log:      
        for line in log:

            # If disconnect is found log it
            if line.find("disconnected") != -1:

            ltime = time.time()

            disconnect = ("Disconnected Found in Log at " + ltime)

            # Check if disconnect is already in the list if not add it
            # to the running list and the msg list      
            if (disconnect not in rec_disconnects):
                    rec_disconnects.append(disconnect)
                    send_disconnects.append(disconnect

Recommended method to pool messages in a open file loop for python

2 Answers2