18

Is there an automated way to do shell piping in Ruby? I'm trying to convert the following shell code to Ruby:

a | b | c... > ...

but the only solution I have found so far is to do the buffer management myself (simplified, untested, hope it gets my meaning across):

a = IO.popen('a')
b = IO.popen('b', 'w+')
Thread.new(a, b) { |in, out|
    out.write(in.readpartial(4096)) until in.eof?
    out.close_write
}
# deal with b.read...

I guess what I'm looking for is a way to tell popen to use an existing stream, instead of creating a new one? Or alternatively, an IO#merge method to connect a's output to b's input? My current approach becomes rather unwieldly when the number of filters grows.

I know about Kernel#system('a | b') obviously, but I need to mix Ruby filters with external program filters in a generic way.

the Tin Man
  • 158,662
  • 42
  • 215
  • 303
Arno
  • 181
  • 1
  • 1
  • 3
  • There is one solution with spawn command in ruby. You should open several pipes and then use spawn options to redirect Stdin and Stdout of child processes. You can find a more detailed example in my answer here: http://stackoverflow.com/questions/11898528/marshal-ruby-pipes-sending-serialized-object-to-child-processes/13258047#13258047 – Pavel Chernov Nov 07 '12 at 19:48
  • Open3's pipelines seem pretty ideal https://ruby-doc.org/stdlib-2.4.1/libdoc/open3/rdoc/Open3.html#method-c-pipeline_rw – kch May 31 '17 at 05:33

5 Answers5

12

Old question, but since its one of the first result on Google, here is the answer : http://devver.wordpress.com/2009/10/12/ruby-subprocesses-part_3/ (method 8)

In short :

sh = Shell.new
sh.system("a") | sh.system("b") | sh.system("c")

And you can do more complicated things like

sh.echo(my_string) | sh.system("wc") > "file_path"
xml = (sh.echo(html) | sh.system("tidy", "-q")).to_s
sloonz
  • 254
  • 2
  • 4
  • Starting from Ruby 2.7, 'shell' is no longer a standard library and requires installing a gem. See https://www.ruby-lang.org/en/news/2019/12/25/ruby-2-7-0-released/ (section 'Other notable changes since 2.6') – knarewski May 13 '22 at 10:25
5

Using plain ruby, spawn has redirection options that you can use to connect processes with pipes.

1) Create a pipe

r,w = IO.pipe

2) Use it to connect two spawned processes

spawn(*%w[echo hello world], out: w)
spawn(*%w[tr a-z A-Z], in: r)
# => HELLO WORLD

Of course, you can encapsulate this in something like sh.system from the mentioned Shell library, and create a |() method for doing the interconnecting.

The open3 module of the standard library has some really nice tools for this kind of stuff, including the creation of complete pipelines.

Petr Skocik
  • 58,047
  • 6
  • 95
  • 142
3

If a, b, and c are commands normally accessed from the command-line then you could use:

captured_output = `a | b | c`

Ruby will run the commands in a sub-shell, and capture STDOUT.

If you need to route the output to a file for some reason, then you can add the redirection to the command too. STDOUT will not be returned to you in that case, but you could open the file and manually process it:

`a | b | c > captured_output`
File.foreach('captured_output') do |li|
  print li
end

It doesn't offer as much control as using system or popen3 but it's convenient:

>> sin, sout, serr = Open3.popen3('ls -al | tail -1') #=> [#<IO:fd 4>, #<IO:fd 5>, #<IO:fd 7>, #<Thread:0x00000100bb8798 run>]
>> sout.read #=> "drwxr-xr-x   3 greg  staff    102 Nov  2 21:01 python\n"
the Tin Man
  • 158,662
  • 42
  • 215
  • 303
  • I would also instead of using Threads made it using big loop for everything ... nasty solution for nasty problem – mpapis Nov 21 '10 at 01:10
-1

Sadly, piping in shell is a serious matter and, indeed, it would require a fair amount of code. It's not necessary to spawn threads with read/write loops, but it would still require a lot of work.

The simplest example I've found is redirect implementation in dash (Debian Almquist Shell). Generally, if you'd want to do the same in Ruby, you'll need to replicate these fd manipulation tricks using Ruby's IO#dup, IO#fileno, IO#pipe, IO#reopen, etc. It might be easier to reuse shell (such as dash's) code in C .so library for Ruby interpreter than trying to combine the same thing using just Ruby primitives.

I don't know of any existing generalized Ruby API for complex interprocess piping / redirection. If you could suggest a good API you'd like to use, may be I could participate in implementation.

GreyCat
  • 16,622
  • 18
  • 74
  • 112
-1

I know this is a super old question, but I just went through this pain.

Basically I came up with a pair of utility methods that run commands with popen within forks, passing results through pipes, to achieve something like:

echo 'ABC12345 234234 24523' | wc | grep '1'

To do it in Ruby:

# Setup two pipes (the two we see in the shell command line above)
# plus one more to receive the result from.
pipe_pair1 = IO.pipe
pipe_pair2 = IO.pipe
pipe_pair_result = IO.pipe
pipe_in_out nil, pipe_pair1,  ['echo', 'ABC12345 234234 24523']
pipe_in_out pipe_pair1, pipe_pair2, ['wc']
pipe_in_out pipe_pair2, pipe_pair_result, ['grep', '1']

# Get the result from the final pipe
puts "Output: #{pipe_result(pipe_pair_result)}"

The code for the methods is on this gist: https://gist.github.com/philayres/c0d96cd263329e41fa84c2e3c7b9ae7b

Phil
  • 2,797
  • 1
  • 24
  • 30