0

I want to get an interval of one hour in my log file. Here's a sample log:

2016-03-30|00:54:46,060|[WARNING]
2016-03-30|00:55:46,318|[OK]
2016-03-30|00:55:46,318|[OK]
2016-03-30|01:42:13,691|[UNKNOWN]
2016-03-30|01:53:16,356|[CRITICAL]
2016-03-30|02:56:41,410|[WARNING]
2016-03-30|02:42:13,691|[UNKNOWN]
2016-03-30|02:53:16,356|[UNKNOWN]
2016-03-30|03:56:41,410|[WARNING]

I define some variables: date_now=date +"%Y-%m-%d %H:%M:%S,%3N", date_minus_one=date -d "-1 hour" +"%Y-%m-%d %H:%M:%S,%3N" and date_minus_two=date -d "-2 hour" +"%Y-%m-%d %H:%M:%S,%3N". I know the logic that you can get a per hour by making `date_now >= date_minus_one and date_now <= date_minus_two. But I don't know how can I make that in awk, sed or grep.

I want the output result will be:

2016-03-30|02:56:41,410|[WARNING]
2016-03-30|02:42:13,691|[UNKNOWN]
2016-03-30|02:53:16,356|[UNKNOWN]

So on and so forth to get a 1 hour interval of logs.

Robert John Martin
  • 41
  • 1
  • 7

5 Answers5

1

Using grep:

grep -E '^2016-03-30\|02:[0-9]{2}:[0-9]{2},' file.log

From a variable:

hour_to_search=$(date '+%F\|%H') 
grep -E "^${hour_to_search}:[0-9]{2}:[0-9]{2}," file.log

Or use a speciic date-time as reference:

hour_to_search=$(date -d '30 Mar 2016 02 AM' '+%F\|%H')
grep -E "^${hour_to_search}:[0-9]{2}:[0-9]{2}," file.log

Example:

$ cat file.log
2016-03-30|00:54:46,060|[WARNING]
2016-03-30|00:55:46,318|[OK]
2016-03-30|00:55:46,318|[OK]
2016-03-30|01:42:13,691|[UNKNOWN]
2016-03-30|01:53:16,356|[CRITICAL]
2016-03-30|02:56:41,410|[WARNING]
2016-03-30|02:42:13,691|[UNKNOWN]
2016-03-30|02:53:16,356|[UNKNOWN]
2016-03-30|03:56:41,410|[WARNING]

$ hour_to_search=$(date -d '30 Mar 2016 02 AM' '+%F\|%H') 

$ echo "$hour_to_search"                                  
2016-03-30\|02

$ grep -E "^${hour_to_search}:[0-9]{2}:[0-9]{2}," file.txt
2016-03-30|02:56:41,410|[WARNING]
2016-03-30|02:42:13,691|[UNKNOWN]
2016-03-30|02:53:16,356|[UNKNOWN]
heemayl
  • 39,294
  • 7
  • 70
  • 76
  • How can i make it dynamic? that every hour I can get the data. No need to enter values of date and hour. Thanks, – Robert John Martin May 03 '16 at 06:12
  • @RobertJohnMartin Check my edits.. – heemayl May 03 '16 at 06:22
  • This approach has the same shortcoming as Michael's answer of being "aligned" on hours of the actual time, so in the worst case there will be almost no output because the hour you are grepping for has just started. – Michael Jaros May 03 '16 at 06:44
  • @heemayl : how about the hour? how can you make it dynamic? I've tried your syntax it works but I need to change the hour to get the past one hour data. – Robert John Martin May 03 '16 at 07:09
  • Nicely done. You can probably assume that after `2016-03-30|02` the format will be consistent and just grep for this part. – fedorqui May 03 '16 at 08:43
0

Make search pattern that includes the date and hour only ("%Y-%m-%d|%H:") and use it for filtering with any one of the tools you've mentioned

GMichael
  • 2,726
  • 1
  • 20
  • 30
  • I've tried awk: `awk '$date_file >= $date_minus_one && $date_file <= $date_minus_two` but it always print all the data. – Robert John Martin May 03 '16 at 06:11
  • I suggested `DATE_PATTERN=$( date +"%Y-%m-%d|%H:"); grep $DATE_PATTERN ` – GMichael May 03 '16 at 06:14
  • This approach will cause the output to be "aligned" on hours, so in the worst case you will get less than one minute of output, because the last hour has just started. – Michael Jaros May 03 '16 at 06:41
  • Yes, but this is exactly what was the purpose of this question: "I want to get an interval of one hour" – GMichael May 03 '16 at 06:43
  • @Michael: (Edit:) Actually I think it's not really clear, the OP could have meant it both ways. – Michael Jaros May 03 '16 at 06:56
  • I do not see another option in this question but I do not want to argue. There are too many ways such an issue can be asked and answered. – GMichael May 03 '16 at 07:02
  • Yes, I want to get data an hour from the past everytime I execute my script. For example I run a crontab job that executes my script per hour and the file output that I want is the data from the past one hour. – Robert John Martin May 03 '16 at 07:05
  • @Robert In this case my answer is good enough. Just form the pattern that meets that last hour. – GMichael May 03 '16 at 07:09
  • @Michael Now I get it I should run my cron 58 or 59 minutes before it become another hour to get 00 to 58 minutes of data. Am I correct? – Robert John Martin May 03 '16 at 07:13
  • Why? This will give you the pattern from the previous hour: `PREV_HOUR=$(date +"%H") ; PREV_HOUR=$(expr $PREV_HOUR - 1) ; DATE_STRING=$( date +"%Y-%m-%d"); DATE_STRING="$DATE_STRING|$PREV_HOUR"` You can optimize it, of course – GMichael May 03 '16 at 07:23
0

You might also try something like

csplit -zk sample.log '/2016-03-30|0'{0..9}/ '{*}'
Michael Vehrs
  • 3,293
  • 11
  • 10
0
#To get the current date and hour



  hour=$( date +'%F|%H')


#Filter the logs 

grep $hour log

Note: Edited Based on suggestion in comment.

monk
  • 1,953
  • 3
  • 21
  • 41
0

Give a try to this

awk -F "|" '{ print "date -d\""$1 " " $2"\" +%s.%N" "'\''|" $0 "'\''" }' mylog.file  |\
  bash |\
  awk -F "|" -v OFS="|" \
    -v startdate=$(date -d "2016-03-30 01:42:13,691" "+%s.%N") \
    -v enddate=$(date -d "2016-03-30 02:53:16,356" "+%s.%N") \
    '$1>=startdate && $1<=enddate {$1="";print}'

Adapt the startdate and enddate to your needs, it should be seconds since EPCOH and nanoseconds: format is "+%s.%N"

First part with awk ... just build a date command line and uses bash to execute it. The goal is to add at the beginning of each line the date as seconds since EPOCH and nanoseconds.

The final awk compares the dates in seconds.nanoseconds format and prints only the lines in the range: [startdate .. enddate]

The test:

$ awk -F "|" \
  '{ print "date -d\""$1 " " $2"\" +%s.%N" "'\''|" $0 "'\''" }' mylog.file  |\
  bash |\
  awk -F "|" -v OFS="|" \
    -v startdate=$(date -d "2016-03-30 01:42:13,691" "+%s.%N") \
    -v enddate=$(date -d "2016-03-30 02:53:16,356" "+%s.%N") \
    '$1>=startdate && $1<=enddate {$1="";print}'

|2016-03-30|01:42:13,691|[UNKNOWN]
|2016-03-30|01:53:16,356|[CRITICAL]
|2016-03-30|02:42:13,691|[UNKNOWN]
|2016-03-30|02:53:16,356|[UNKNOWN]
Jay jargot
  • 2,745
  • 1
  • 11
  • 14