3

I want to monitor a log file, when a new log message match my defined pattern (say contain “error”), then send out an email to me.

To do that, I wrote a python script monitor.py, the main part looks like:

import sys

for line in sys.stdin:
    if "error" in line:
        print line

It works well when I use tail my.log | python monitor.py, then I switch to tail -f my.log | python monitor.py, then it doesn’t work, at least not immediately.

I have done some tests, when the new content to the log accumulate up to 8KB, then my python script can get output from tail. So I highly suspect that this is controlled by the stdin/stdout buffer size. How can I get the output immediately?

One more question, when I use tail -f my.log and tail -f my.log | grep error, why it could show me the output immediately?

Wenzhong
  • 56
  • 7

2 Answers2

5

Most Linux programs will use line buffering if stdout is connecting to a TTY and full buffering otherwise. You can use stdbuf to force line buffering. 

stdbuf -oL tail -f my.log | python monitor.py
John Kugelman
  • 349,597
  • 67
  • 533
  • 578
  • Looking at [man stdbuf](http://linux.die.net/man/1/stdbuf), are we supposed to prefix `stdbuf` to `tail -f` or `python` command? or both are OK? – anishsane Mar 04 '14 at 06:37
  • Thank you John, this is absolutely the right direction! But I still fail to make it work as expect, does it work for you? My stdbuf version is 8.4 and according to my blank terminal, I think my python script still can not read output from `tail -f` even using `stdbuf -oL` – Wenzhong Mar 04 '14 at 06:40
  • @anishsane , you get any luck? – Wenzhong Mar 05 '14 at 02:42
0

There's a patch to add unbuffered output to tail, dating from 2008. which appears to have been rejected and my own (BSD) manpage does not indicate it. Perhaps you could download coreutils, apply the patch, compile tail yourself and it may still work?

hd1
  • 33,938
  • 5
  • 80
  • 91