5

Hi I am trying to tail apache access logs and copy the errors to another file. I tried below options and all are working in command line but when triggered from a script they are not working.

I understand the tail command is not exiting and so there is no output. But not sure how to overcome this.

/usr/bin/tail -f /apps/apache/logs/access_log | grep -h "HTTP\/1.1\" 50." >> /tmp/log_error_capture.txt
 grep -m 1 "HTTP\/1.1\" 50." <(tail -f /apps/apache/logs/access_log)
( tail -f -n0 /apps/apache/logs/access_log & ) | grep -q "HTTP\/1.1\" 50." > /tmp/log_error_capture.txt
tail -f logfile |grep -m 1 "HTTP\/1.1\" 50." | xargs echo "" >> logfile \;

Can someone suggest a better way to grep the errors. Please.

fedorqui
  • 275,237
  • 103
  • 548
  • 598
  • 1
    try adding `--line-buffered` option to grep. – anishsane May 23 '16 at 05:27
  • we often had such situations so its always advisable to check every 10secs, tail of log file, mark the line till which you are done. so next time again tail last lines based on the mark you did, till end of file. do let me know if you didnt get me. – Sanjeev May 23 '16 at 05:39
  • 1
    If you want `tail -f`you don't *want* the command to exit. If you just want to grep the files as of right now, you don't need tail, just grep. Maybe you want to background with `&`? Please clarify. – tripleee May 23 '16 at 05:46
  • Hi Triplee, This is a script continuously monitoring the access log and copy the lines with errors to another file so we need to monitor the file. –  May 23 '16 at 06:25
  • Hi Sanjeev, Yes I thought of it but how to manage during log rotation. –  May 23 '16 at 06:27
  • Hi Anishsane, I tried --line-buffered but its not working and I am using the below version of tail. tail (GNU coreutils) 8.4 [apache@RHEL01 tmp]$ tail -f --line-buffered /apps/apache/logs/access_log | grep "HTTP\/1.1\" 50." tail: unrecognized option '--line-buffered' Try `tail --help' for more information. –  May 23 '16 at 06:28
  • I do see lot many people raised question on same topic but not sure what they did for their situation. –  May 23 '16 at 06:31

1 Answers1

8

If you want to monitor the growing log file, there is no way for your command to complete until the log file stops growing. I'm guessing you simply want to background the job:

tail -f access_log | grep "HTTP/1\.[01]\" 50." >> /tmp/log_error_capture.txt &

Incidentally, I removed the backslash before the slash (the slash is not a regex special, so it doesn't require escaping) and added one before the dot (where the opposite holds) and also updated the expression to accept HTTP/1.0 as well as 1.1. (I've occasionally seen 0.9, too, but I guess those are just testing and/or negligible anyway.) Also, because grep is reading standard input, the -h option was useless, so I removed it.

This will continue to run, but not produce anything useful, when the log file is rotated. Perhaps you want to keep the PID and restart it as part of your log rotation script; or perhaps you simply want to run the grep once at log rotation, instead of having it keep grinding in the background.

Alternatively, you could use tail --follow=name to keep on running even across log rotations. For a production system, you should still figure out how to cope with log rotation for this script (i.e. how and when to rotate the file of 50x errors).

tripleee
  • 175,061
  • 34
  • 275
  • 318
  • Hi Triplee, I am wondering the above redirection to the file is not working, I tried some generic redirects and all works fine but the above one is working only when I grep it without output redirection :((. –  May 23 '16 at 09:40
  • http://stackoverflow.com/questions/30787575/using-tail-f-on-a-log-file-with-grep-in-bash-script?rq=1 Here larsks mentioned that it wont work. I think same happening for me. –  May 23 '16 at 09:46
  • The below is working finally. tail -f /apps/apache/logs/access_log | grep --line-buffered "HTTP/1\.[01]\" 50." | while read line do echo $line >> /tmp/log_error_capture.txt done –  May 23 '16 at 10:39
  • The `while` loop is just obfuscation, and you lack the proper quotes in the `echo` (and for good measure you should use `read -r`). You are experiencing buffering. This is a very common FAQ. http://mywiki.wooledge.org/BashFAQ/009 – tripleee May 23 '16 at 11:20
  • Hi Tripleee, Can you please explain a bit how while loop is not good. –  May 24 '16 at 08:50
  • 1
    If it does something useful, surely adding more `while` loops will be even more useful? `cat file | cat | cat | while read -r line; do echo "$line" | while read -r line2;` ... – tripleee May 24 '16 at 09:24
  • Hi, I understand your concern but here I do not see any other option working other than a while loop. I corrected with read -r and added quotes as you have mentioned. Thank you very much for your patience. –  May 24 '16 at 10:38
  • Your call, obviously; but it isn't really necessary, and if you perceive at as an improvement to disable buffering, there are much superior ways to fix that, most of which are also syntactically much simpler and hence less error-prone. – tripleee May 24 '16 at 10:48