2

For the purposes of publishing metrics to AWS CloudWatch I would like to get information of the number of occurrences of some keyword (Eg., Error, Exception) within the last minute (from current system time) in my application logs.

Following are the commands that I have tried so far based on the answers from a related thread ( Filter log file entries based on date range):

awk -vDate=`date -d'now-1 minutes' +["%Y-%m-%d %H:%M:%S"` '($1 FS $2) > Date {print $3}' application.log | grep "ERROR" | uniq -c

awk -vDate=`date -d'now-1 minutes' +["%Y-%m-%d %H:%M:%S"` '{if ($1 > Date) {print $3}}' application.log | grep "ERROR" | uniq -c

awk -vDate=`date -d'now-1 minutes' +["%Y-%m-%d %H:%M:%S"` '{if ($1 == $Date) {print $3}}' application.log | grep "ERROR" | uniq -c

But I get an error like this when I try this:

awk: cmd. line:1: 13:06:17
awk: cmd. line:1:   ^ syntax error

Following is the format of my log file:

2016-02-05 12:10:48,761 [INFO] from org.xxx
2016-02-05 12:10:48,761 [INFO] from org.xxx
2016-02-05 12:10:48,763 [INFO] from org.xxx
2016-02-05 12:10:48,763 [INFO] from org.xxx
2016-02-05 12:10:48,763 [ERROR] from org.xxx
2016-02-05 12:10:48,763 [INFO] from org.xxx
2016-02-05 12:10:48,764 [INFO] ffrom org.xxx
2016-02-05 12:10:48,773 [WARN] from org.xxx
2016-02-05 12:10:48,777 [INFO] from org.xxx
2016-02-05 12:10:48,778 [INFO] from org.xxx

Stuck on this for quite a while. Thanks for the help!

Community
  • 1
  • 1
nitarshs
  • 173
  • 2
  • 10

1 Answers1

0

You're using deprecated backticks and so not quoting the date output. Do this instead:

awk -vDate="$(date -d'now-1 minutes' +"%Y-%m-%d %H:%M:%S")" '($1 FS $2) > Date { if ($3~/ERROR/) print $3}' file

Note that you don't need to pipe to grep and by not having a space between -v and Date your script is gawk-specific and if it's gawk-specific then you don't need that external call to date since gawk has it's own builtin time functions (hint: BEGIN{Date=strftime("%Y-%m-%d %H:%M:%S",systime()-60)}).

You also don't need uniq -c but without seeing your real input and expected output (doing a uniq -c given that input wouldn't make any sense vs wc -l) I'm not going to guess any more.

Oh what the heck, here's the whole script in gawk:

$ cat tst.awk
BEGIN {
    #date = strftime("%Y-%m-%d %H:%M:%S",systime()-60)
    date = "2016-02-05 12:10:48"
}
($1" "$2) > date {
    if ($3 ~ /ERROR/) {
        cnt[$3]++
    }
}
END {
    for (err in cnt) {
        print err, cnt[err]
    }
}
$
$ awk -f tst.awk file
[ERROR] 1

I assume in reality you have various flavors of "ERROR" and that's why you want the count of each. Just uncomment the strftime line and delete the hard-coded timestamp line to run on your real data.

Ed Morton
  • 188,023
  • 17
  • 78
  • 185
  • Now, I no longer get that error. But I don't get anything else either. (I tried for "INFO" string since they are available within the last minute). Thanks – nitarshs Feb 05 '16 at 14:06
  • You are doing something wrong. I recommend you try it with a fixed `Date` value to debug. – Ed Morton Feb 05 '16 at 14:07
  • Yes, even I am trying without `uniq -c` to see if it is able to get some output at the very least. – nitarshs Feb 05 '16 at 14:09
  • It's completely unclear why you are using `uniq -c`. It makes no sense at all to do so given that input file and that script. You DO know you had a spurious `[` at the start of your Date assignment don't you? I removed that in my answer. – Ed Morton Feb 05 '16 at 14:11
  • Ok, I am trying using fixed date now. Yes, I am using `uniq -c` hoping to get the count of the number of times I get the ERROR message in the last minute. Yes, I picked it up from the command in the linked thread. I did not know what it was for, so I did not remove it. – nitarshs Feb 05 '16 at 14:20
  • I edited my answer to show the full gawk version of the script. Just change the `cnt` array to a scalar if you don't have different flavors of error. – Ed Morton Feb 05 '16 at 14:21
  • Yes, I using AWS CloudWatch to publish this metric information every minute. like this: `aws cloudwatch put-metric-data --metric-name ErrorCount --namespace "MyService" --value 2` . So I would substitute the value argument with that obtained from your command. Also, Sorry I am unable to upvote your answer since I do not have enough reputation yet. :( – nitarshs Feb 05 '16 at 14:35