Ruby IO.popen STDOUT buffering

Question

I'm working on a script that uses IO.popen to open another program and continually read the data. It's like this:

process = IO.popen(["/the/program", "argument", "argument"])

loop do
  line = process.gets
  puts "#{line}"
end

(The actual program does more than just printing the output, obviously - that's just an example.)

The issue I'm running into is that popen seems to be buffering STDOUT from the opened process. I've confirmed this by running the program directly from a shell and through popen, side-by-side, and the Ruby one never gets one line at a time. It always gets multiple lines at a time, and is delayed.

I've tried

STDOUT.sync = true

... before popen, but that hasn't changed anything.

The program in question is definitely using \n as a new line, so that's not the issue.

Did you try `process.sync = true`? – DNNX Aug 19 '11 at 05:55 — DNNX, Aug 19 '11 at 05:55
@Eddie did you figure this out? – Travis Reeder Dec 18 '12 at 01:39 — Travis Reeder, Dec 18 '12 at 01:39

score 6 · Answer 1 · edited May 23 '17 at 11:54

6

Do you have the source to the other program? You either need to force the other program to flush its output, or make your script look like a pty (see the pty standard lib).

See this question for a good explanation of what's going on.

EDIT: pty sample code:

require 'pty'
PTY.spawn "some-command" do |r,w,p|
  loop { puts r.gets }
end

edited May 23 '17 at 11:54

Community

1
1

answered Aug 19 '11 at 05:47

cam

14,192
1
44
29

This fails to print the last line if it doesn't end in a newline (e.g. if some-command is echo -n ABC). – Paul Brannan Apr 20 '17 at 18:59

score 2 · Answer 2 · edited May 23 '17 at 12:32

I suspect /the/program is buffering when it detects stdout is not a terminal - you can test by piping through cat, eg:

"/the/program" "argument" "argument" | cat

The answer above, will solve it if that is the problem, ie:

#!/usr/bin/env ruby

require 'pty'
PTY.spawn "./the-program testing one Two three" do |r,w,p|
  loop { puts "GOT: #{r.gets}" }  
end

Some languages (eg C) detect if stdout is a terminal and change to line buffered - see Is stdout line buffered, unbuffered or indeterminate by default?

As an example when it works, I used a simple bash script to output each argument and the time, one at a time, with 3 seconds in between between and the ruby script worked without a problem. I added eof detection for this example.

Modified script:

#!/usr/bin/env ruby

process = IO.popen(["./the-program", "testing", "one", "Two", "three"])

while !process.eof?
  line = process.gets
  puts "GOT: #{line}"
end

the-program contents:

#!/bin/bash

for arg
do
  echo $arg
  date
  sleep 3
done

I tried with ruby version 1.9.3 and 2.1.2

$ ruby ,p
GOT: testing
GOT: Mon Jun 16 06:19:00 EST 2014
GOT: one
GOT: Mon Jun 16 06:19:03 EST 2014
GOT: Two
GOT: Mon Jun 16 06:19:06 EST 2014
GOT: three
GOT: Mon Jun 16 06:19:09 EST 2014
$

If I use a C program instead, then the problem reoccurs:

#include <stdio.h>

main(int argc, char **argv)
{
        int i;

        for (i=0; i<argc; i++) {
                printf("%s\n", argv[i]);
                sleep(3);
        }
}

Ruby IO.popen STDOUT buffering

2 Answers2