103

How can I pipe several external commands together in Go? I've tried this code but I get an error that says exit status 1.

package main

import (
    "io"
    "log"
    "os"
    "os/exec"
)

func main() {
    c1 := exec.Command("ls")
    stdout1, err := c1.StdoutPipe()
    if err != nil {
        log.Fatal(err)
    }

    if err = c1.Start(); err != nil {
        log.Fatal(err)
    }
    if err = c1.Wait(); err != nil {
        log.Fatal(err)
    }

    c2 := exec.Command("wc", "-l")
    c2.Stdin = stdout1

    stdout2, err := c2.StdoutPipe()
    if err != nil {
        log.Fatal(err)
    }

    if err = c2.Start(); err != nil {
        log.Fatal(err)
    }
    if err = c2.Wait(); err != nil {
        log.Fatal(err)
    }

    io.Copy(os.Stdout, stdout2)
}
Kevin Burke
  • 61,194
  • 76
  • 188
  • 305

8 Answers8

164

For simple scenarios, you could use this approach:

bash -c "echo 'your command goes here'"

For example, this function retrieves the CPU model name using piped commands:

func getCPUmodel() string {
        cmd := "cat /proc/cpuinfo | egrep '^model name' | uniq | awk '{print substr($0, index($0,$4))}'"
        out, err := exec.Command("bash","-c",cmd).Output()
        if err != nil {
                return fmt.Sprintf("Failed to execute command: %s", cmd)
        }
        return string(out)
}
Stefan Saru
  • 1,731
  • 2
  • 10
  • 4
  • 1
    It should be noted, however, that this will fail if the cmd is too long. In particular if it's over 131072 bytes long, then you'll probably get something like `fork/exec /bin/bash: argument list too long`, see [here](https://stackoverflow.com/questions/11475221/why-do-i-get-bin-sh-argument-list-too-long-when-passing-quoted-arguments/11475732#11475732). In this case you might have change or divide up your command, or resort to the more hefty `io.Pipe` methods listed elsewhere in this question's answers. – henrywallace Jan 31 '19 at 19:45
  • 1
    There is another key takeaway from this question/answer. What a 'command' is in Go? It stands for an executable, and not a 'shell command', as one might expect. So, the command here is `bash`, with an option (`-c`) and a 'shell command' argument. One could argue that `bash` may not be available on the system, and that's much more likely than a 100KB 'command' to break this solution. A bunch of pipes and buffers + dozen lines of code to collect a one-liner shell command output (that even no longer reads as a one-liner), is plain unacceptable. I think this should be the accepted one. – tishma Sep 07 '19 at 13:03
  • 2
    This should be the simplest answer, even though it depends on `bash`. That's good! – Marcello DeSales Oct 22 '19 at 23:22
  • I should note CombinedOutput() is probably better than Output() in most cases,as it includes the STDERR output from the program, so that you can see if an error happened, instead of having silent errors – Felipe Valdes Apr 15 '21 at 02:58
  • 1
    There will be a newline char as part of the `Output()` stored as the last byte in `out`. It can be stripped off by re-slicing it, i.e. `out = out[:len(out)-1]` – Inian Jun 03 '21 at 08:19
  • Doing so may introduce a security issue if your command arguments rely on user input and appropriate care is not taken to sanitize them (aka shell injection) – kucherenkovova Nov 16 '22 at 13:37
69

StdoutPipe returns a pipe that will be connected to the command's standard output when the command starts. The pipe will be closed automatically after Wait sees the command exit.

(from http://golang.org/pkg/os/exec/#Cmd.StdinPipe )

The fact you do c1.Wait closes the stdoutPipe.

I made a working example (just a demo, add error catching!) :

package main

import (
    "bytes"
    "io"
    "os"
    "os/exec"
)

func main() {
    c1 := exec.Command("ls")
    c2 := exec.Command("wc", "-l")

    r, w := io.Pipe() 
    c1.Stdout = w
    c2.Stdin = r

    var b2 bytes.Buffer
    c2.Stdout = &b2

    c1.Start()
    c2.Start()
    c1.Wait()
    w.Close()
    c2.Wait()
    io.Copy(os.Stdout, &b2)
}
Denys Séguret
  • 372,613
  • 87
  • 782
  • 758
  • 6
    Why do use io.Pipe rather than exec.Cmd.StdoutPipe? –  May 28 '12 at 10:02
  • 1
    I like io.Pipe too, but putting the c1 start into a separate goroutine works better for me. See my modified version below. – WeakPointer Mar 12 '16 at 15:25
  • @WeakPointer When to use `os.Pipe()`? because `io.Pipe()` is performing IPC without any issue, in the above code – overexchange Oct 09 '20 at 20:30
  • @overexchange Sorry, don't understand the question. It's been years since I looked at this stuff heavily but they have very different signatures don't they? os.Pipe connects one *os.File to another. io.Pipe() returns two items, one can can do io.Read on a byte slice, and one can do io.Write on a byte slice. – WeakPointer Oct 10 '20 at 17:17
  • @WeakPointer Am confused with return types of `os.Pipe()` vs `io.Pipe()`. `os.Pipe()` returns `File*` and the documentation says, `Pipe returns a connected pair of Files; reads from r return bytes written to w. It returns the files and an error, if any.` So, How is this different from `io.Reader` & `io.Writer` that `io.Pipe()` returns? – overexchange Oct 10 '20 at 17:38
61
package main

import (
    "os"
    "os/exec"
)

func main() {
    c1 := exec.Command("ls")
    c2 := exec.Command("wc", "-l")
    c2.Stdin, _ = c1.StdoutPipe()
    c2.Stdout = os.Stdout
    _ = c2.Start()
    _ = c1.Run()
    _ = c2.Wait()
}
Matt
  • 1,424
  • 10
  • 15
  • 1
    I'm basically using the same code but often I get a "broken pipe" error. Any idea what could cause that? http://stackoverflow.com/q/26122072/4063955 – Anthony Hunt Oct 05 '14 at 18:06
  • @AnthonyHat: please put this comment on your new question, so we can see you saw this one and it didn't work for you. – RickyA Oct 06 '14 at 10:23
  • Broken pipe occurs when one process tries to write into a pipe but the other side of the pipe has already been closed. For example, if the "wc -l" exited before the "ls" finished in the above example, the "ls" would get a Broken Pipe error/signal. – Matt Oct 06 '14 at 22:13
  • How can i make this program concurrent, since there is io happening here(stdin/stdout)!! – user7044 Feb 22 '15 at 19:41
  • 4
    @user7044, I'm not sure what you mean. The two commands "ls" and "wc -l" are running concurrently in this example, with the output from ls being piped to wc, which can start reading the output from ls before ls finishes writing all of it. – Matt May 19 '15 at 14:50
  • 2
    This answer does not seem to be correct. Documentation says "For the same reason, it is incorrect to call Run when using StdoutPipe." See https://pkg.go.dev/os/exec#Cmd.StdoutPipe Probably that also explains @AnthonyHunt's broken pipes. – Socci Apr 05 '21 at 13:18
  • @Socci, I don't think that's true. The pipe created is an OS level pipe. The subprocess running 'ls' won't close it's handle on write end of the pipe until it's done. The Go program closes it's handle on the write end just after 'ls' starts, but that's fine because it's been dup'd into the subprocess. – Matt Apr 06 '21 at 19:32
  • @AnthonyHunt's question is really about stopping goroutines, and doesn't mention pipes or subprocesses at all. – Matt Apr 06 '21 at 19:35
9

Like the first answer but with the first command started and waited for in a goroutine. This keeps the pipe happy.

package main

import (
    "io"
    "os"
    "os/exec"
)

func main() {
    c1 := exec.Command("ls")
    c2 := exec.Command("wc", "-l")

    pr, pw := io.Pipe()
    c1.Stdout = pw
    c2.Stdin = pr
    c2.Stdout = os.Stdout

    c1.Start()
    c2.Start()

    go func() {
        defer pw.Close()

        c1.Wait()
    }()
    c2.Wait()
}
WeakPointer
  • 3,087
  • 27
  • 22
  • 2
    It would probably work just fine without the goroutine if it used os.Pipe() instead of io.Pipe(). Let the OS do the byte-shuffling by itself. – Jason Stewart Jul 26 '18 at 18:29
  • 1
    @JasonStewart This advice seems to be correct, thx. I've begun using it with no ill effects so far. – WeakPointer Feb 10 '19 at 15:22
  • @WeakPointer When to use `os.Pipe()`... if `io.Pipe()` can perform IPC?http://www.albertoleal.me/posts/golang-pipes.html – overexchange Oct 09 '20 at 20:26
7

This is a fully working example. The Execute function takes any number of exec.Cmd instances (using a variadic function) and then loops over them correctly attaching the output of stdout to the stdin of the next command. This must be done before any function is called.

The call function then goes about calling the commands in a loop, using defers to call recursively and ensuring proper closure of pipes 

package main

import (
    "bytes"
    "io"
    "log"
    "os"
    "os/exec"
)

func Execute(output_buffer *bytes.Buffer, stack ...*exec.Cmd) (err error) {
    var error_buffer bytes.Buffer
    pipe_stack := make([]*io.PipeWriter, len(stack)-1)
    i := 0
    for ; i < len(stack)-1; i++ {
        stdin_pipe, stdout_pipe := io.Pipe()
        stack[i].Stdout = stdout_pipe
        stack[i].Stderr = &error_buffer
        stack[i+1].Stdin = stdin_pipe
        pipe_stack[i] = stdout_pipe
    }
    stack[i].Stdout = output_buffer
    stack[i].Stderr = &error_buffer

    if err := call(stack, pipe_stack); err != nil {
        log.Fatalln(string(error_buffer.Bytes()), err)
    }
    return err
}

func call(stack []*exec.Cmd, pipes []*io.PipeWriter) (err error) {
    if stack[0].Process == nil {
        if err = stack[0].Start(); err != nil {
            return err
        }
    }
    if len(stack) > 1 {
        if err = stack[1].Start(); err != nil {
             return err
        }
        defer func() {
            if err == nil {
                pipes[0].Close()
                err = call(stack[1:], pipes[1:])
            }
        }()
    }
    return stack[0].Wait()
}

func main() {
    var b bytes.Buffer
    if err := Execute(&b,
        exec.Command("ls", "/Users/tyndyll/Downloads"),
        exec.Command("grep", "as"),
        exec.Command("sort", "-r"),
    ); err != nil {
        log.Fatalln(err)
    }
    io.Copy(os.Stdout, &b)
}

Available in this gist

https://gist.github.com/tyndyll/89fbb2c2273f83a074dc

A good point to know is that shell variables like ~ are not interpolated

Tyndyll
  • 334
  • 3
  • 5
3
package main

import (
    ...
    pipe "github.com/b4b4r07/go-pipe"
)

func main() {
    var b bytes.Buffer
    pipe.Command(&b,
        exec.Command("ls", "/Users/b4b4r07/Downloads"),
        exec.Command("grep", "Vim"),
    )

    io.Copy(os.Stdout, &b)
}

I spent a good day trying to use Denys Séguret answer to come up with a wrapper for multiple exec.Command before I came across this neat package by b4b4r07.

eriel marimon
  • 1,230
  • 1
  • 19
  • 29
  • I just realized the implementation of this package is the same as the answer by @Tyndyll up above. Just noting... – eriel marimon Sep 30 '18 at 17:44
  • 1
    I don't know, maybe it's quite obvious to everybody but it wasn't as obvious to me and I learned the hard way that when you actually do io.Copy() call at the end you won't get the result because it's already at &b :) – toudi Jul 10 '20 at 10:53
2

I wanted to pipe some video and audio to FFplay. This worked for me:

package main

import (
   "io"
   "os/exec"
)

func main() {
   ffmpeg := exec.Command(
      "ffmpeg", "-i", "247.webm", "-i", "251.webm", "-c", "copy", "-f", "webm", "-",
   )
   ffplay := exec.Command("ffplay", "-")
   ffplay.Stdin, ffmpeg.Stdout = io.Pipe()
   ffmpeg.Start()
   ffplay.Run()
}

https://golang.org/pkg/io#Pipe

Zombo
  • 1
  • 62
  • 391
  • 407
1

Because it can be complex to build such command chains I have decided to implements a litte go library for that purpose: https://github.com/rainu/go-command-chain

package main

import (
    "bytes"
    "fmt"
    "github.com/rainu/go-command-chain"
)

func main() {
    output := &bytes.Buffer{}

    err := cmdchain.Builder().
        Join("ls").
        Join("wc", "-l").
        Finalize().WithOutput(output).Run()

    if err != nil {
        panic(err)
    }
    fmt.Printf("Errors found: %s", output)
}

With the help of this lib you can also configure std-error forwarding and other things.

rainu
  • 11
  • 1