1

I have to make a shell script such that each time it is executed it goes to the log file which is common and gets updated. What is already being done is pick all the erroneous logs and show as output. But now I want this to be like it picks logs which is certain minutes older and no more.

grep is good for fetching the timestamp line. But how do I do it to get only fresh data and not too old.

grep -i error log.log |  grep `date +%Y'-'%m'-'%d`

Log file format is as follows : '[date in YYYY-mm-dd timestamp] - error/exception generated'

Any help will be appreciated. Thank you all in advance.

Preeti Maurya
  • 431
  • 1
  • 7
  • 17
  • 1
    Show what have you tried so far ? And provide few examples too.. – blackSmith Apr 20 '16 at 12:21
  • hi @blackSmith. provided example. Thanks for looking – Preeti Maurya Apr 20 '16 at 12:24
  • By examples I actually meant a sample file, never mind. And fetching logs based on timestamps is really tricky to achieve with regexes. – blackSmith Apr 20 '16 at 12:29
  • 1
    sorry @blackSmith cant add that. but i will add the format of the log file. – Preeti Maurya Apr 20 '16 at 12:30
  • How can I store the value of `date --date='60 minutes ago'` in another shell script temp variable. This will be helpful maybe @blackSmith – Preeti Maurya Apr 20 '16 at 13:54
  • 1
    `tmp=$(date --date='60 minutes ago' +%s)` – David C. Rankin Apr 21 '16 at 01:25
  • 1
    post a [mcve] so we can try to help you. – Ed Morton Apr 21 '16 at 02:54
  • Possible duplicate of [extract data from log file in specified range of time](http://stackoverflow.com/questions/7575267/extract-data-from-log-file-in-specified-range-of-time) – tripleee Apr 21 '16 at 04:22
  • @DavidC.Rankin - the `--date` option may work in Linux, but we don't know that the OP us running Linux. We need more data before we can provide answers. – ghoti Apr 21 '16 at 05:37
  • Since it was tagged as `Unix` it is fairly safe that a comparable `date` function would, or could, be available. If you look at comment `5` , you will see a specific request for how to save time generated with the `--date` option to a `temp` variable. That too was a dead giveaway. – David C. Rankin Apr 21 '16 at 05:44

1 Answers1

2

This is somewhat tricky. The date of 60 minutes ago would help grep for entries that occurred exactly 60 minutes ago, but wouldn't help find things after that. Luckily, awk is good with timestamps:

awk -F']' -v limit="[`date -d '60 minutes ago' +'%Y-%m-%d %H:%M:%S'`" '
    BEGIN {IGNORECASE=1} /error/ && $1 > limit' log.log

Note the use of -F']' to set the delimiter to ], and the addition of the [ at the beginning of the timestamp limit string. This means awk will actually be comparing timestamps including their opening bracket, e.g., "[2016-04-20 17:55:08", but that's ok, it still works.

Don't like awk? How about:

start=`date -d '60 minutes ago' '%Y-%m-%d %H:%M:%S'`
sort -d <(grep -i error log.log) <(echo "[$start") |
  sed -n "/$start/,$p" |
  grep -v "^\[$start$"
webb
  • 4,180
  • 1
  • 17
  • 26
  • 1
    The `grep` is completely unnecessary and [useless](http://www.iki.fi/era/unix/award.html#grep). It's much more efficient to do `awk -options... '/error/ && ($1 > limit)' log.log` – tripleee Apr 21 '16 at 04:23
  • 1
    Alas, awk is NOT good with timestamps. The `strftime()` and `systime()` functions exist only in `gawk` as far as I'm aware. If you don't know the OP's platform, it's best either to ask for clarification or to make your answer portable. – ghoti Apr 21 '16 at 05:42
  • +1 for the awk solution. It's not possible with only grep, although hour level handling can be done if it's in 24hr format. – blackSmith Apr 21 '16 at 07:36
  • 1
    thank you so much @webb. I will try and tell if this works for me. – Preeti Maurya Apr 21 '16 at 13:02