Golang pipe output of subcommand real time

Question

I am trying to pipe the output of a command, but no data seems to be read from the pipe until the write end is closed. Eventually I want this to connect to a websocket that streams the status of a command while it is being executed. The problem is that while this code prints the messages line by line, it does not print anything until the program has finished executing.

cmd := exec.Command(MY_SCRIPT_LOCATION, args)

// create a pipe for the output of the script
// TODO pipe stderr too
cmdReader, err := cmd.StdoutPipe()
if err != nil {
    fmt.Fprintln(os.Stderr, "Error creating StdoutPipe for Cmd", err)
    return
}

scanner := bufio.NewScanner(cmdReader)
go func() {
    for scanner.Scan() {
        fmt.Printf("\t > %s\n", scanner.Text())
    }
}()

err = cmd.Start()
if err != nil {
    fmt.Fprintln(os.Stderr, "Error starting Cmd", err)
    return
}

err = cmd.Wait()
if err != nil {
    fmt.Fprintln(os.Stderr, "Error waiting for Cmd", err)
    return
}

Is there any way I can do something similar and have a scanner actually read line by line as it is written to the pipe instead of after everything has been written? The program takes about 20 seconds to run, and there is a steady stream of updates, so it is annoying to have them all go through at once.

Related: [Streaming commands output progress](http://stackoverflow.com/questions/30725751/streaming-commands-output-progress) — icza, Jun 18 '15 at 19:33
@icza thanks for pointing me there, but unfortunately it does not help much. The person there has pretty much the same setup as I do. At first I thought the problem was that it couldn't read the newlines in my file, but the `scanner.Scan()` loop is executed once per line of the output, so the scan delimiter should work fine — mrosales, Jun 18 '15 at 20:24
have you tried just connecting `cmd.StdoutPipe` to `os.Stdout`? — Not_a_Golfer, Jun 18 '15 at 22:28

score 6 · Accepted Answer · answered Jun 18 '15 at 22:36

6

Turns out that the issue was not in the code I posted above. That works as expected. The problem was that the C program that was being executed was not properly flushing stdout. When running it interactively, it worked as expected, but when stdout was piped, it would not actually get written until I called flush. After manually adding some flush statements to the c program, the go code worked as expected.

answered Jun 18 '15 at 22:36

mrosales

1,543
11
18

1

Thanks coming back and writing this answer after figuring it out! I was losing my mind for this! I was similar problem only with python script. – briiC Jan 15 '17 at 11:56

score 0 · Answer 2 · answered Jul 03 '21 at 12:36

F.Y.I

read line by line as it is written to the pipe instead of after everything has been written?

I found the package go-pipeline very useful for those who came to this topic by googling.

github.com/mattn/go-pipeline @ GitHub

The below is equivalent to git log --online | grep first import | wc -l

package sample

import (
    "fmt"
    "log"

    "github.com/mattn/go-pipeline"
)

func ExampleCommandPipeLine() {
    out, err := pipeline.Output(
        []string{"git", "log", "--oneline"},
        []string{"grep", "first import"},
        []string{"wc", "-l"},
    )
    if err != nil {
        log.Fatal(err)
    }

    fmt.Println(string(out))
    // Output:
    // 1
}

View other sample online @ Go Playground

Golang pipe output of subcommand real time

2 Answers2

Linked