Log parsing in Python: Alert if more than N errors in a given period?

Question

I have a log file that contains an hour's worth of data.

I'd like to generate an alert if there are more than N errors recorded within any 5-minute period within that log.

What I don't want to do is designate 5-minute periods (e.g. 00-05, 06-10, etc.) and iterate through them, because if N=10, and I have 8 errors at 04 and 8 errors at 07, they'll be treated as in two separate buckets and won't generate the alert.

I suppose I could instead iterate 60 times, advancing 1 minute each time, and consider the 5-minute bucket from that point, but is there a more elegant or more performant way?

Answer 1

I would use a sliding window (see Rolling or sliding window iterator in Python for reference) over the list of errors and then check for every iteration if first and last entry are within 5 minutes

example (from Rolling or sliding window iterator in Python):

from collections import deque

def window(seq, n=2):
    it = iter(seq)
    win = deque((next(it, None) for _ in xrange(n)), maxlen=n)
    yield win
    append = win.append
    for e in it:
        append(e)
        yield win

for w in window(errors, 10):
    # if (w[-1]['timestamp'] - w[0]['timestamp']) > 60*5:
    #     error

Answer 2

I decided to take the advice in depperm's comment (and I wish it had been submitted as an answer rather than a comment so I could mark it as accepted).

It looks something like this:

error_queue = []
max_errors = 3
for log_line in log_lines:
    log_ts = get_timestamp(log_line)
    if contains_error(log_line):
        error_queue.append(log_ts)
        interval_start = log_ts - datetime.timedelta(minutes=5)
        try:
            threshold = error_queue[-max_errors]
        except IndexError:
            continue
        if threshold and threshold >= interval_start:
            raise Exception