How to search for lines in a file between two timestamps using Bash

Question

In bash I am trying to read a log file and will print only the lines that have a timestamp between two specific times. The time format is hh:mm:ss. For example, I would be searching for lines that would fall between 12:52:33 to 12:59:33.

I want to use regular expression because I can use it in grep function.

Each log line begins with some_nr 2014-05-15 21:58:00,000000 rest_of_line.

My solution gives me lines with 1 min margin. I cut out ss and take all lines with hh:mm:[0-9]{2}. $2 has format filename_hh:mm:; for example: "24249_16:05:;24249_16:05:;24249_16:07:;24249_16:07:;24249_16:08:"

My code:

B=$2  

for line in ${B//;/ } ;
do  
    TENT=`echo $line | awk '{split($0,numbers,"_"); print numbers[1]}'`"_logs.txt"
    TIME=`echo $line | awk '{split($0,numbers,"_"); print numbers[2]}'`"[0-9]{2}"

    grep -iE ${TIME} ${TENT} >> ${FILE1}
done

I need a solution with 15 sec margin for any time not 60. I want to have input in format filename_hh:mm:ss and take lines for hh:mm:ss +/- 15s or filename_hh:mm:ss(1)_hh:mm:ss(2) and take lines between hh:mm:ss(1) and hh:mm:ss(2). For sometime there is no lines so the solution should 'recognize' if sometimes match inputted interval or not.

Log files look like this:

1002143 1002143 2014/15/05 22:09:52.937004 bla 
1002130         2014/15/05 22:09:44.786002 bla bla
1001667         2014/15/05 22:09:44.592009 bl a bla
1001667 1001667 2014/15/05 22:09:44.592009 bl a bla

Can you post a sample of your log file? – anubhava May 16 '14 at 15:48 — anubhava, May 16 '14 at 15:48
yes, I added in in question above. – herder May 16 '14 at 15:55 — herder, May 16 '14 at 15:55

score 18 · Accepted Answer · answered May 16 '14 at 15:01

18

I believe sed is the best option:

sed -rne '/<timestamp>/,/<timestamp>/ p' <file>

ex:

tiago@dell:~$ sed -rne '/08:17:38/,/08:24:36/ p' /var/log/syslog 
May 16 08:17:38 dell AptDaemon.Worker: INFO: Processing transaction /org/debian/apt/transaction/08a244f7b8ce4fad9f6b304aca9eae7a
May 16 08:17:50 dell AptDaemon.Worker: INFO: Finished transaction /org/debian/apt/transaction/08a244f7b8ce4fad9f6b304aca9eae7a
May 16 08:18:50 dell AptDaemon.PackageKit: INFO: Initializing PackageKit transaction
May 16 08:18:50 dell AptDaemon.Worker: INFO: Simulating trans: /org/debian/apt/transaction/37c3ef54a6ba4933a561c49b3fac5f6e
May 16 08:18:50 dell AptDaemon.Worker: INFO: Processing transaction /org/debian/apt/transaction/37c3ef54a6ba4933a561c49b3fac5f6e
May 16 08:18:51 dell AptDaemon.PackageKit: INFO: Get updates()
May 16 08:18:52 dell AptDaemon.Worker: INFO: Finished transaction /org/debian/apt/transaction/37c3ef54a6ba4933a561c49b3fac5f6e
May 16 08:24:36 dell AptDaemon: INFO: Quitting due to inactivity

answered May 16 '14 at 15:01

Tiago Lopo

7,619
1
30
51

no sed is definitely not the "best" option. think about that the log output could have timestamp – Kent May 16 '14 at 15:03
1

@kent in that case one could match the beggining of the line – Tiago Lopo May 16 '14 at 15:10
my log lines are beginning with: 2014-05-15 21:58:00,000000 and becouse those miliseconds I wanted to use grep. – herder May 16 '14 at 15:28
herder would you mind to share which grep command are you using? – Tiago Lopo May 16 '14 at 15:36
I add my code in question above. – herder May 16 '14 at 15:42
sorry, my log lines are beginning with: 1002143 2014/15/05 22:09:52.937004 – herder May 16 '14 at 17:59
It doesn't work if time is not in first column. In my case I discover one more problem: sometimes time is in third column and sometimes in fourth... so I think grep is the only possible solution. – herder May 20 '14 at 07:30
@TiagoLopo My log file can include that time stamp in messege itself like "2022-03-10T14:00 {"responsetime: 2022-03-10T16:59.7877+878"} in utc format or as json. as you can see timstap is 2022-03-10T14:00..but if I search for 2022-03-10T16:59 as start time it also includes those data. How can I not search inside the message ? – Khurshid Alam Mar 10 '22 at 14:02
@KhurshidAlam match the beginning of the line like `sed -rne '/^May 16 08:17:38/,/^May 16 08:24:36/ p' /var/log/syslog` – Tiago Lopo Mar 10 '22 at 16:11

score 10 · Answer 2 · edited May 09 '17 at 00:01

10

log file is usually sorted by timestamp, assume the timestamp is on the first column, you could:

awk -v from="12:52:33" -v to="12:59:33" '$1>=from && $1<=to' foo.log

in this way, you can change the from and to to get different set of log entries. regex is not a good tool to do number calculation/comparison.

edited May 09 '17 at 00:01

A. Wilson

8,534
1
26
39

answered May 16 '14 at 14:42

Kent

189,393
32
233
301

This solution also doesn't work. In my log_file in second column sometimes is a number and sometimes not - and above code doesn't recognize properly columns. If in second column there is a number then it see time in 4th, if not - then in 3rd. – herder May 20 '14 at 07:39
@herder the solution was for the 1st version of your question. You edited it and added new stuff, the solution will fail. but awk has date related function, also it can invoke external command. it could do your job. read some tutorials. – Kent May 20 '14 at 07:54
I appreciate your help - I know it was for 1st version of question but I didn't predict those problems. I'll search in man awk ;) – herder May 20 '14 at 08:55
awk -v from=$TIME1".000" -v to=$TIME2".000" '{for (i=3; i<=4; i++) if ($i~"([0-9]{2}:){2}[0-9]{2}.[0-9]{3}" && ($i<=to && $i>=from)) print $0}' $TENT – herder May 20 '14 at 13:20
I used if to check in which column I have a time format. But actually using perl is more comfortable. Thanks everyone for help. – herder May 20 '14 at 13:27

score 4 · Answer 3 · answered May 16 '14 at 14:37

4

You can use this regex in egrep:

egrep '12:5[2-9]:33' file.log

answered May 16 '14 at 14:37

anubhava

761,203
64
569
643

I need general solution, not only for this example but any two times. – herder May 16 '14 at 15:22
Sure that is also possible but 1. It will be better to clarify this in your question and 2. Show your own attempt. – anubhava May 16 '14 at 15:24
I added my tries in question above. – herder May 16 '14 at 15:40

Stefan Schmiedl · Answer 4 · 2014-05-16T17:23:25.753

2

You are using the wrong tool for this task. Once you have a regular expression like the one given by @anubhava, you can easily find a time interval that is not matched by it. grep and regexps might work for a few special cases, but they do not scale to the general case.

Can you use some tool that can actually "understand" the timestamps? Probably every scripting language out there (perl, python, ruby, lua) has builtin or library support for parsing time and date.

However, you might be able to employ the powers of GNU date:

% date --date="2014-05-15 21:58:00 15 sec ago" +'%Y-%m-%d %H:%M:%S'
2014-05-15 21:57:45
% date --date="2014-05-15 21:58:00 15 sec" +'%Y-%m-%d %H:%M:%S' 
2014-05-15 21:58:15

and plug that into Tiago's sed filter idea.

edited May 16 '14 at 17:23

answered May 16 '14 at 14:43

Stefan Schmiedl

505
5
21

you are right, but this is not an answer buddy. – Kent May 16 '14 at 14:44
sorry, just couldn't help myself ... – Stefan Schmiedl May 16 '14 at 15:06

score 1 · Answer 5 · answered May 16 '14 at 19:32

1

You can try the following perl script:

#! /usr/bin/perl

use warnings;
use strict;
use Time::Piece;
use autodie;

my $arg=shift;
my @a =split("_",$arg);
my $fn=shift @a;

my $dfmt='%Y/%d/%m';
my $fmt=$dfmt.' %H:%M:%S';
my $t = localtime;
my $date=$t->strftime($dfmt);
my $t1; my $t2;
if (@a == 1) {
   my $d=$date.' '.$a[0];
   my $tt=Time::Piece->strptime($d, $fmt);
   $t1=$tt-15;
   $t2=$tt+15;
} elsif (@a == 2) {
   $t1=Time::Piece->strptime($date.' '.$a[0], $fmt);
   $t2=Time::Piece->strptime($date.' '.$a[1], $fmt);
} else {
   die "Unexpected input argument!";
}

$fn=$fn.'_logs.txt';
doGrep($fn,$t1,$t2,$fmt);

sub doGrep { 
   my ($fn,$t1,$t2,$fmt) = @_;

   open (my $fh, "<", $fn);
   while (my $line=<$fh>) {
      my ($d1,$d2) = $line=~/\S+\s+(\S+)\s+(\d\d:\d\d:\d\d)/;
      my $d=$d1.' '.$d2;
      my $t=Time::Piece->strptime($d, $fmt);
      print $line if ($t>$t1 && $t<$t2);
   }
   close ($fh);
}

Run it from command line using syntax : ./p.pl A_22:09:14.

answered May 16 '14 at 19:32

Håkon Hægland

39,012
21
81
174

I got an error: "Error parsing time at C:/Perl64/lib/Time/Piece.pm line 469, <$_[...]> line 1." – herder May 20 '14 at 08:42
@herder Maybe you could paste an example of the logfile you are using on http://pastebin.com/ ? Then I could try to run it on my machine also.. to check – Håkon Hægland May 20 '14 at 08:53
@HH - thanks! http://pastebin.com/Us6BrbSF – herder May 20 '14 at 09:18
1

@herder Thanks.. It seems that your file has a different date format than that you gave in your question.. Compare `04/16/14` and `2014/15/05` – Håkon Hægland May 20 '14 at 09:26
1

@herder In your question you only specified a time interval. There was no requirement regarding the date. So the script assumed it was the current date. However, it seems that the log file is not from the current date. What would you like to do? Specify a date on the command line also? – Håkon Hægland May 20 '14 at 09:33
@HH sorry. I want to ignore it. – herder May 20 '14 at 10:35
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/54028/discussion-between-hakon-haegland-and-herder). – Håkon Hægland May 20 '14 at 10:35
@herder I moved the discussion to chat. Please click link above. – Håkon Hægland May 20 '14 at 10:44

How to search for lines in a file between two timestamps using Bash

5 Answers5

Linked