15

I am getting unexpected behaviour using popen3, which I want to use to run a command like tool ala cmd < file1 > file2. The below example hangs, so that stdout done is never reached. Using other tools than cat may cause hanging, so that stdin done is never reached. I suspect, I am suffering from buffering, but how do I fix this? 

#!/usr/bin/env ruby

require 'open3'

Open3.popen3("cat") do |stdin, stdout, stderr, wait_thr|
  stdin.puts "foobar"

  puts "stdin done"

  stdout.each_line { |line| puts line }

  puts "stdout done"

  puts wait_thr.value
end

puts "all done"
Adam Spiers
  • 17,397
  • 5
  • 46
  • 65
maasha
  • 1,926
  • 3
  • 25
  • 45

3 Answers3

15

stdout.each_line is waiting for further output from cat because cat's output stream is still open. It's still open because cat is still waiting for input from the user because its input stream hasn't been closed yet (you'll notice that when you open cat in a terminal and type in foobar, it will still be running and waiting for input until you press ^d to close the stream).

So to fix this, simply call stdin.close before you print the output.

sepp2k
  • 363,768
  • 54
  • 674
  • 675
  • Thanks. I did test this, and it works nicely on this example. However on my real-life stuff it still hangs, but that is also using threads (and using temporary files instead of popen3 works fine). Perhaps popen3 is not thread-safe? – maasha Jan 21 '12 at 10:03
  • 3
    This fix works only if the argument to `stdin.puts` is short. If you write much, much more data to the pipe associated with your variable `stdin`, `cat` will write to the buffer of the pipe for stdout until the buffer is full. At this moment, the OS will suspend `cat`. If you write more data to your `stdin` in this situation, you fill the buffer of the stdin pipe. When this pipe is full, calling `stdin.puts` blocks. This means: Your program hangs. So if you have much data, you have to interleave writing to `stdin` and reading from `stdout` (and `stderr`). Why not use `Open3.capture*`? – hagello Sep 04 '15 at 20:03
  • @maasha `popen3` is thread-safe. Though you need profound knowledge of inter-process communication to use it right in real life scenarios. – hagello Sep 04 '15 at 20:08
7

Your code is hanging, because stdin is still open!

You need to close it with IO#close or with IO#close_write if you use popen3.

If you use popen then you need to use IO#close_write because it only uses one file descriptor.

 #!/usr/bin/env ruby
 require 'open3'

 Open3.popen3("cat") do |stdin, stdout, stderr, wait_thr|
   stdin.puts "foobar"

   stdin.close   # close stdin like this!  or with stdin.close_write

   stdout.each_line { |line| puts line }

   puts wait_thr.value
 end

See also:

Ruby 1.8.7 IO#close_write

Ruby 1.9.2 IO#close_write

Ruby 2.3.1 IO#close_write

Tilo
  • 33,354
  • 5
  • 79
  • 106
  • This, of course, depends on whether or not the process you are piping to writes to stdout upon an EOT from stdin. If the process writes to stdout upon every line it gets, then it's possible that something else is causing the hang. – Ten Bitcomb Aug 29 '16 at 17:25
7

The answers by Tilo and by sepp2k are right: If you close stdin, your simple test will end. Problem solved.

Though in your comment to the answer of sepp2k, you indicate that you still experience hangs. Well, there are some traps that you might have overlooked.

Stuck on full buffer for stderr

If you call a program that prints more to stderr than the buffer of an anonymous pipe can hold (64KiB for current Linuxes), the program gets suspended. A suspended program neither exits nor closes stdout. Consequently, reading from its stdout will hang. So if you want to do it right, you have to use threads or IO.select, non-blocking, unbuffered reads in order to read from both stdout and stderr in parallel or by turns without getting stuck.

Stuck on full buffer for stdin

If you try to feed more (much more) than "foobar" to your program (cat), the buffer of the anonymous pipe for stdout will get full. The OS will suspend cat. If you write even more to stdin, the buffer of the anonymous pipe for stdin will get full. Then your call to stdin.write will get stuck. This means: You need to write to stdin, read from stdout and read from stderr in parallel or by turns.

Conclusion

Read a good book (Richards Stevens, "UNIX Network Programming: Interprocess communications") and use good library functions. IPC (interprocess communications) is just too complicated and prone to indeterministic run-time behavior. It is for too much hassle to try to get it right by try-and-error.

Use Open3.capture3.

hagello
  • 2,843
  • 2
  • 27
  • 37