Python - Print all log lines that match a pattern between 2 time stamps

Question

Working to print lines that match a specific pattern between a start time and an end time.

Example log line:

Fri Mar 01 HH:MM:SS YYYY:thresholdcrossed

I am looking for an equivalent of the following:

cat /path/file.log | egrep -v "exec|param" | grep thresholdcrossed | sed -n /$start/,/$stop/ >output.file

The start and stop will be user generated variables based on the start and stop times they are searching.

This will give me matches for a single line using regular expressions, but I can't get all lines that match between 2 time stamps.

for line in hand:
    line = line.rstrip()
    if re.search("Mar 10 21.+:.+tstat-threshcrosso", line)  :
        print line

When you saw "_but I can't get lines that match between 2 delimiters_", could you elaborate? What are the delimiters you mention? — cs95, Jun 12 '17 at 19:45
I need all lines that match tstat-threshcrosso between a start date/time and stop date/time There are lots of extra between each threshcrosso over match that doesn't need to be captured. — ausrakay, Jun 12 '17 at 19:48
Sorry. I am trying to figure out how to format my existing code — ausrakay, Jun 12 '17 at 19:49
@Suni D S Yes. The data looks like this when matching the regex string. Fri Mar 10 21:51:07 2017:tstat-threshcrosso Fri Mar 10 21:51:15 2017:tstat-threshcrosso Fri Mar 10 21:55:38 2017:tstat-threshcrosso However between each line wanted there are many more that aren't required — ausrakay, Jun 12 '17 at 19:53
post the potential values for `$start` and `$stop` variables — RomanPerekhrest, Jun 12 '17 at 20:44

Sunil D S · Answer 1 · 2017-06-12T20:27:04.150

0

# user defined time range in milliseconds since epoch and
start_ms = 
end_ms = 
# user defined pattern
pattern = "tstat-threshcrosso"

# loop through whole file
for line in hand:
    # strip the line
    line = line.rstrip()

    # parse the line to get time_stamp_ms (ms since epoch) and log_message
    time_stamp_ms = 
    log_message =

    # break if we went beyond end time
    if time_stamp_ms > end:
        break

    # check if the time stamp is within user defined period
    if time_stamp_ms < start_ms:
        continue

    # check for string pattern match
    if pattern in log_message:
        print line

You can optimize this by doing a binary search to find the first line after the start time.

edited Jun 12 '17 at 20:27

answered Jun 12 '17 at 19:56

Sunil D S

130
11

Would the code change if I needed all that matched tstat-threshcrosso" Between 2 different times example: Mar 10 21 to Mar 10 23 – ausrakay Jun 12 '17 at 20:00
@ausrakay define "all" – Sunil D S Jun 12 '17 at 20:01
Match all lines that contain tstat-threshcrosso between a start and stop times – ausrakay Jun 12 '17 at 20:06
This piece of code does find you **all** lines between start and stop that match your pattern. – Sunil D S Jun 12 '17 at 20:07
Sorry for my ignorance. How would I enter milliseconds since epoch? – ausrakay Jun 12 '17 at 20:14
https://currentmillis.com/ - be careful with the timezone. https://stackoverflow.com/questions/10494312/parsing-time-string-in-python – Sunil D S Jun 12 '17 at 20:17
Sorry, but I was pulled away from the problem. – ausrakay Aug 14 '17 at 18:59

Python - Print all log lines that match a pattern between 2 time stamps

1 Answers1