7

I'm trying to write a ruby script to filter the output of a tailed file (tail -f log.log | ./my_filter.rb). I believe I've set stdin and stdout to be read synchronously, but I still see my output coming out in delayed batches, 20 or so lines at a time, rather than in realtime.

I can reproduce the problem with code as simple as:

#!/usr/bin/ruby     
                
$stdout.sync = true 
$stdin.sync = true  
                
ARGF.each do |line| 
  puts line         
end

Am I missing a setting to eliminate buffering, or something along those lines?

Edit: To clarify, if I just tail -f the log then I see many lines written per second.

JellicleCat
  • 28,480
  • 24
  • 109
  • 162
Michael
  • 462
  • 1
  • 6
  • 18
  • 2
    And if you do just `tail -f log.log`, is it coming in realtime then? – Petr Skocik Oct 12 '14 at 20:12
  • 1
    How is the log file being created? I suspect that whatever process is creating it is buffering its output, so your Ruby script only sees the chunks as they are written. – matt Oct 12 '14 at 20:25
  • It is the OS itself that buffers your data as it passes the pipe. – Rein Oct 12 '14 at 21:43
  • When I just `tail -f` the log, updates stream in realtime, many lines per second, so I think the problem is with my script. If I tail the log into `grep` or another command line utility everything is realtime as well. – Michael Oct 12 '14 at 23:00
  • two non-ruby solutions: http://unix.stackexchange.com/a/297672/14907 – akostadinov Jul 22 '16 at 21:24

3 Answers3

4

If you're dealing with files, you probably want IO#fsync, which says:

Immediately writes all buffered data in ios to disk. Note that fsync differs from using IO#sync=. The latter ensures that data is flushed from Ruby’s buffers, but does not guarantee that the underlying operating system actually writes it to disk.

If you're just dealing with standard input and output, you might also try requiring io/console to see if using IO::console#ioflush gives you the behavior you need. The documentation says:

Flushes input and output buffers in kernel. You must require ‘io/console’ to use this method.

As an example, consider:

require 'io/console'

ARGF.each do |line|
  $stdout.puts line
  $stdout.ioflush
end
Todd A. Jacobs
  • 81,402
  • 15
  • 141
  • 199
  • 2
    This doesn't really answer the question. Question is how to make random output from executed program non-buffered or line buffered at most. Adding calls after each puts is rather non-feasible and non-desirable. – akostadinov Jul 22 '16 at 20:06
  • 1
    @akostadinov Old, I know, but `$stdout.sync = true` should do the trick. – Frizlab Apr 23 '20 at 10:17
3

This is an old question, but the accepted answer does not fully answer the question IMHO.

You can put $stdout.sync = true at the beginning of the program to get sync output to stdout (with a require 'io/console' if needed).

Frizlab
  • 846
  • 9
  • 30
1

The title of this thread includes stdin as well as stdout, so adding to the other answers, none of which addresses stdin:

$stdin.iflush # Discard anything currently in $stdin's byte buffer
JellicleCat
  • 28,480
  • 24
  • 109
  • 162