0

I want to get the data between two times in a log file of different months and date.Suppose if my startime is not present in the logfile, then I want to extract the data from the nearest next time in the logfile. And also it has to end before the endtime, if the entered endtime is not present in the log file.

My log file data,

Apr 10 16 02:07:20  Data 1
Apr 11 16 02:07:20  Data 1
May 10 16 04:11:09  Data 2
May 12 16 04:11:09  Data 2
Jun 11 16 06:22:35  Data 3
Jun 12 16 06:22:35  Data 3

The solution I am using is,

awk -v start="$StartTime" -v stop="$EndTime" 'start <= $StartTime && $EndTime <= stop' $file

where, I am storing my starttime in $StartTime and endtime in $EndTimeBut Iam not getting the exact output. Please help.

anishsane
  • 20,270
  • 5
  • 40
  • 73
Vedh
  • 93
  • 1
  • 10
  • See: [How to filter data between 2 dates with awk in a bash script](http://stackoverflow.com/q/28275880/3776858) – Cyrus Jul 05 '16 at 05:41
  • 1
    Your `awk` solution is one option, and most likely the most flexible, but you can also implement a solution with a simple `while` loop in bash checking and parsing each line in the log file. Which every path you take, when you get stuck, just edit your question here and people are happy to help. – David C. Rankin Jul 05 '16 at 05:45
  • The date in your logs is not sorted in _right_ way, when compared to the source of your script (Assuming that you got the `awk` script from some other SO question). `awk`'s `<=` or `>=` would work for string (using string compare operations) or numbers (arithmetic comparison). Here, you have used `$StartTime` & `$Endtime`. Since `StartTime` & `Endtime` are uninitialized in awk script, so, `start` & `stop` are compared against `$0`, i.e. entire line. – anishsane Jul 05 '16 at 05:58
  • Thanks for the guidelines.What is the best option to get the output? – Vedh Jul 05 '16 at 06:12
  • Convert `"Apr 10 16 02:07:20"` sequence to `YYYYMMDDhhmmss` with optional colon/dash/slashess in between. e.g. above case could be changed to `2016/04/10-02:07:20` – anishsane Jul 05 '16 at 06:21
  • ^^ Note: You don't have to change the format of the log file. You can parse the timestamp in awk & convert it to format in above comment. – anishsane Jul 05 '16 at 06:28
  • if I convert to `2016/04/10-02:07:20` format, then Is it possible to get the output from the above code? And also, can I use `$StartTime` variable or hard coded value? – Vedh Jul 05 '16 at 06:40

1 Answers1

1

Something like this maybe:

$ BashVarStart="16 05 10 00 00 00" # the same format that awk function will reformat to
$ BashVarStop="16 06 11 00 00 00"
$ awk -v start="$BashVarStart" -v stop="$BashVarStop" -F"[ :]" -v OFS=\  '
function reformatdate(m,d,y,h,mm,s) { # basically throw year to the beginning
  monstr="Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec";   # numerize the months
  split(monstr,monarr," ");           # split monstr to an array to enumerate the months
                                      # monarr[1]="Jan", monarr[2]="Feb" etc
  for(i in monarr) {                  # iterate over all month numbers in monarr index
    if(monarr[i]==m)                  # when month number matches
      m=sprintf("%02d",i)             # zeropad if month number below 10: 9 -> 09
  }; 
  return y" "m" "d" "h" "mm" "s       # return in different order   
} 
start < reformatdate($1,$2,$3,$4,$5,$6) && stop > reformatdate($1,$2,$3,$4,$5,$6)
' test.in
May 10 16 04:11:09  Data 2
May 12 16 04:11:09  Data 2
James Brown
  • 36,089
  • 7
  • 43
  • 59
  • Thanks for the answer. May I know, `test.in` is the log file name? If yes, I used `log.sh` as filename and I got error like `awk: cmd. line:1: log.sh awk: cmd. line:1: ^ syntax error`.Any idea about the reason for the error? – Vedh Jul 05 '16 at 09:41
  • Maybe you should cut the AWK program to its own file (this.awk) and call it from your script like: `awk -v start="$BashVarStart" -v stop="$BashVarStop" -F"[ :]" -v OFS=\ -f this.awk test.in`. AWK program is the part between single quotes ('). Forget those quotes from the AWK program file. – James Brown Jul 05 '16 at 09:48
  • ya I inserted the function in a `.sh` file and called it like `awk -v start="$BashVarStart" -v stop="$BashVarStop" -F"[ :]" -v OFS=\ -f new.sh log.sh` And also I used it in `.awk` format.But still the same issue. – Vedh Jul 05 '16 at 09:53
  • There should be 2 spaces after `-v OFS=\ `. First one is the value of OFS (\ ) and the next one separates the switched and their parameters. – James Brown Jul 05 '16 at 10:01
  • Thanks.. When I use the code, the result contains the data after the StartTime. How can I include the starttime also in the result? I have used `start <= reformatdate($1,$2,$3,$4,$5,$6) && stop > reformatdate($1,$2,$3,$4,$5,$6)` like this.. But still I cannot get the StartTime. – Vedh Jul 07 '16 at 05:43
  • It was acting funny when I forgot the `-F"[ :]" -v OFS=\ ` part but otherwise it works. I just tested it including and excluding the limiting dates. Try to debug it and add a `print "DEBUG: "y" "m" "d" "h" "mm" "s` before the return. It should output something like - -`DEBUG: 16 05 12 04 11 09` `May 12 16 04:11:09 Data 2`- -. – James Brown Jul 07 '16 at 06:10
  • Ok.. Suppose if my log file has `\[11\/04\/16 00:00:00 BST\]` format and I also want the output in `\[11\/04\/16 00:00:00 BST\]` format with the same input format, where the code has to be changed? in `return` part? – Vedh Jul 07 '16 at 06:33
  • I'd change the input field separator to -F"[ :]" to allow more separator characters, like [ and /. Are the backslashes in the data? – James Brown Jul 07 '16 at 06:43
  • Shall I post it as a new question? – Vedh Jul 07 '16 at 06:46
  • It would be easier. And please post some more sample data. – James Brown Jul 07 '16 at 06:58
  • `awk -v start="$BashVarStart" 'BEGIN{FS="[[\\\\/: ]+"} start >= $4" "$3" "$2" "$5" "$6" "$7 ' test.in` assuming the backslashes in your example are in data, not escapes and that the date format is dd\/mm\/yy hh:mm:ss and discarding the TZ. – James Brown Jul 07 '16 at 07:07
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/116656/discussion-between-vishwaroopa-and-james-brown). – Vedh Jul 07 '16 at 08:51