ruby pipes, IO and stderr redirect

Question

I am looking to have a ruby program (a rake task) observe an output from another rake task. The output writer outputs to stderr. I'd like to read those lines. I'm having difficulty setting it up. If I have a writer (stdout_writer.rb) that constantly prints something:

#!/usr/bin/env ruby
puts 'writing...'
while true
  $stdout.puts '~'
  sleep 1
end

and a file that reads it and echoes (stdin_reader.rb):

#!/usr/bin/env ruby
puts 'reading...'
while input = ARGF.gets
  puts input
  input.each_line do |line|
    begin
      $stdout.puts "got line #{line}"
    rescue Errno::EPIPE
      exit(74)
    end
  end
end

and I'm trying to have them work together, nothing happens:

$ ./stdout_writer.rb 2>&1 | ./stdin_reader.rb
$ ./stdout_writer.rb | ./stdin_reader.rb

nothing... although if I just echo into stdin_reader.rb, I get what I expect:

piousbox@e7440:~/projects/opera_events/sendgrid-example-operaevent$ echo "ok true" | ./stdin_reader.rb
reading...
ok true
got line ok true
piousbox@e7440:~/projects/opera_events/sendgrid-example-operaevent$

so how would I setup a script that gets stderr piped into it, so that it can read it line-by-line? Additional info: this will be an ubuntu upstart service script1.rb | script2.rb where script1 sends a message, and script2 verifies that the message was sent by script1

score 1 · Answer 1 · answered Oct 19 '16 at 01:55

The issue seems to be that as stdout_writer runs infinitely, stdin_reader will never get a chance to read the STDOUT from stdout_writer as the pipe, in this case, is waiting for stdout_writer to be finished before stdin_reader starts reading. I tested this by changing while true to 5.times do. If you do that, and wait 5 seconds, the result of ./stdout_writer.rb | ./stdin_reader.rb is

reading...
writing...
got line writing...
~
got line ~
~
got line ~
~
got line ~
~
got line ~
~
got line ~

This isn't an issue with your code itself, but more so an issue with the way that ruby execution in terms of STDOUT | STDIN handling works.

Also, I don't think I've ever learned as much as I learned researching this question. Thank you for the fun exercise.

score 1 · Answer 2 · answered Oct 19 '16 at 02:36

The output from stdout_writer.rb is being buffered by Ruby, so the reader process doesn’t see it. If you wait long enough, you should see the result appear in chunks.

You can turn buffering off and get the result you’re expecting by setting sync to true on $stdout at the start of stdout_writer.rb:

$stdout.sync = true

ruby pipes, IO and stderr redirect

2 Answers2

Linked