How can I iterate over the last 5 mins of my log file using bash

Question

I have huge xferlog and I'm trying to parse it. My log file like this:

Wed Nov  5 16:41:36 2014 1 10.8.0.6 0 /home/spy/16.41.txt b _ i r spy ftp 0 * c
Wed Nov  5 16:42:07 2014 1 10.8.0.6 0 /home/spy/16.42.txt b _ i r spy ftp 0 * c
Wed Nov  5 16:43:45 2014 1 10.8.0.6 0 /home/spy/16.43.txt b _ i r spy ftp 0 * c
Wed Nov  5 16:44:34 2014 1 10.8.0.6 0 /home/spy/16.44.txt b _ i r spy ftp 0 * c
Wed Nov  5 16:45:57 2014 1 10.8.0.6 0 /home/spy/16.45.txt b _ i r spy ftp 0 * c
Wed Nov  5 16:46:52 2014 1 10.8.0.6 0 /home/spy/16.46.txt b _ i r spy ftp 0 * c
Wed Nov  5 16:46:55 2014 1 10.8.0.6 0 /home/spy/16.46.2.txt b _ i r spy ftp 0 * c
Wed Nov  5 16:47:23 2014 1 10.8.0.6 0 /home/spy/16.47.txt b _ i r spy ftp 0 * c

My bash script reads the log file(while all line) and does something. I need to run my script for the without last 5 mins of my log file(not last few lines, because I don't know how many line).

How can I do?

Bash script like this:

 #!/bin/bash

 while read -r ...
 do
 ... 
 done < $LOG

My second question, how can I pass lines that in last 5 minutes interval:

Log:

Wed Nov  5 16:41:36 2014 1 10.8.0.6 0 /home/spy/16.41.txt b _ i r spy ftp 0 * c
Wed Nov  5 16:42:07 2014 1 10.8.0.6 0 /home/spy/16.42.txt b _ i r spy ftp 0 * c
Wed Nov  5 16:43:45 2014 1 10.8.0.6 0 /home/spy/16.43.txt b _ i r spy ftp 0 * c
Wed Nov  5 16:44:34 2014 1 10.8.0.6 0 /home/spy/16.44.txt b _ i r spy ftp 0 * c
Wed Nov  5 16:44:59 2014 1 10.8.0.6 0 /home/spy/16.44.txt b _ i r spy ftp 0 * c
Wed Nov  5 16:45:00 2014 1 10.8.0.6 0 /home/spy/16.45.txt b _ i r spy ftp 0 * c
Wed Nov  5 16:46:52 2014 1 10.8.0.6 0 /home/spy/16.46.txt b _ i r spy ftp 0 * c
Wed Nov  5 16:46:55 2014 1 10.8.0.6 0 /home/spy/16.46.2.txt b _ i r spy ftp 0 * c
Wed Nov  5 16:47:23 2014 1 10.8.0.6 0 /home/spy/16.47.txt b _ i r spy ftp 0 * c

Last line 16:47:23. I need to pass 16:45:00-16:49:59 lines. I need to print:

Wed Nov  5 16:41:36 2014 1 10.8.0.6 0 /home/spy/16.41.txt b _ i r spy ftp 0 * c
Wed Nov  5 16:42:07 2014 1 10.8.0.6 0 /home/spy/16.42.txt b _ i r spy ftp 0 * c
Wed Nov  5 16:43:45 2014 1 10.8.0.6 0 /home/spy/16.43.txt b _ i r spy ftp 0 * c
Wed Nov  5 16:44:34 2014 1 10.8.0.6 0 /home/spy/16.44.txt b _ i r spy ftp 0 * c
Wed Nov  5 16:44:59 2014 1 10.8.0.6 0 /home/spy/16.44.txt b _ i r spy ftp 0 * c

So when I have new line like this:

Wed Nov  5 16:50:00 2014 1 10.8.0.6 0 /home/spy/16.50.txt b _ i r spy ftp 0 * c

I need to pass 16:50:00-16:54:59.

possible duplicate of [extract last 10 minutes from logfile](http://stackoverflow.com/questions/20649387/extract-last-10-minutes-from-logfile) — aphex, Nov 06 '14 at 13:16
I can't understand answers on this topic. I don't want to show last 5 or 10 mins. I don't want to run for the last 5 mins. I want to run before last 5 mins. — onur, Nov 06 '14 at 14:36
Hi volkanasr, explain more about your scenario and your requirement and what does "before last 5 mins" mean ,it means from last 5-10 mins back ? — Sriharsha Kalluru, Nov 06 '14 at 14:49
5 mins mean before 5 mins of last log time. For example on my post last line 16:47:21. I need to skip between this lines --> 16:42 - 16:47:21. — onur, Nov 06 '14 at 15:16

glenn jackman · Accepted Answer · 2014-11-06T16:05:41.797

2

Since you say your file is quite large, you don't want to parse and call date for every line, that would be quite slow. That leaves parsing the file with a language that has date parsing facilities. Here's some perl:

perl -MTime::Piece -MTime::Seconds -Mautodie -e '
    sub entry_time {
        Time::Piece->strptime(substr(shift, 0, 24),"%a %b %e %T %Y");
    }

    $filename = shift;
    $last_line = qx{ tail -n1 $file };
    $last_time = entry_time $last_line;
    $five_minutes_ago = $last_time - 5*ONE_MINUTE;

    open $fh, "<", $filename;
    while (<$fh>) {
        $time = entry_time $_;
        last if $time > $five_minutes_ago;
        print;
    }
    close $fh;
' xferlog

with bash

entry_time() {
    date -d "$(cut -c 1-24 <<< "$1")" +%s
}

LOG=xferlog
cutoff=$(( $(entry_time "$(tail -n1 "$LOG")") - 5*60 ))

while IFS= read -r line; do
    t=$(entry_time "$line")
    (( t > cutoff )) && break
    echo "$line"
done < "$LOG"

edited Nov 06 '14 at 16:05

answered Nov 06 '14 at 14:59

glenn jackman

238,783
38
220
352

Thank you but sorry for wrong understanding because of edited my first post. I dont need last 5 mins of my log file. I need all line without last 5 mins of my log file. I need read last line time and back to 5 mins and dont do anything of last 5 min files. For example on my post last line 16:47:21. I need to skip between this lines --> 16:42 - 16:47:21. – onur Nov 06 '14 at 15:14
Thanks, so I can't do this with just bash :)? – onur Nov 06 '14 at 15:57
1

You can with bash. I bet it will be a lot slower than the perl. – glenn jackman Nov 06 '14 at 16:05
Thank you, your bash sample works for me. I have a second question, edited my first post. Sometimes I need not pass last 5 min, I need pass last 5 mins interval(like 16:45:00-16:50:00). Can I do this with like your sample? – onur Nov 09 '14 at 18:35
I try to use `cutoff=$(( $(entry_time "$(tail -n1 "$LOG")") - $(entry_time "$(tail -n1 "$LOG")") % (5 * 60) ))` – onur Nov 10 '14 at 08:41

How can I iterate over the last 5 mins of my log file using bash

1 Answers1