Using io.Pipe to get output from exec.command to make http post

Question

Was reading up on the use of io.Pipe to reduce allocation (no need to assign cmd.Stdout to bytes.Buffer). I couldn't get the following to work and appreciate if greatly on how can i use io.Pipe to get output from exec.command to make http mutliform post request.

func main() {
    cmd := exec.Command("convert", "test.png", "-resize" "30x30", "png:-")
    bodyReader, bodyWriter := io.Pipe()
    cmd.Stdout = bodyWriter
    formWriter := multipart.NewWriter(bodyWriter)
    go func() {
        cmd.Run()
        partWriter, err := formWriter.CreateFormFile("file", "file")
        if err != nil {
            fmt.Println("form writer create error")
            return
        }
        _, err = io.Copy(partWriter, bodyReader)
        if err != nil {
            fmt.Println("io copy error")
            return
        }
        formWriter.Close()
        bodyWriter.Close()
    }()

    url := "http://example.com/upload"
    req, _ := http.NewRequest(http.MethodPost, url, bodyReader)
    req.Header.Set("Content-Type", formWriter.FormDataContentType())
    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        fmt.Println("htttp do error")
        return
    }
    fmt.Println(resp.StatusCode)
    respBody, err := ioutil.ReadAll(resp.Body)
    if err != nil {
        fmt.Println("read error")
        return
    }
    fmt.Println("Result:", string(respBody))
}

Using debugging tool, it just just hang at resp, err := http.DefaultClient.Do(req) and eventually the following output.

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x1218c3b]

goroutine 1 [running]:
main.main()

Which line of code in the program panicked? – Oct 10 '21 at 05:33 — , Oct 10 '21 at 05:33

score 0 · Answer 1 · answered Oct 10 '21 at 06:03

You need to add another Pipe for multipart.Writer

func main() {
    cmd := exec.Command("convert", "./test.jpg", "-resize", "30x30", "jpg:-")
    bodyReader, bodyWriter := io.Pipe()
    cmd.Stdout = bodyWriter
    cmd.Stderr = os.Stderr
    formReader, formWriter := io.Pipe()
    multiWriter := multipart.NewWriter(formWriter)

    go func() {
        if err := cmd.Run(); err != nil {
            bodyWriter.CloseWithError(err)
        } else {
            bodyWriter.Close()
        }
    }()

    go func() {
        partWriter, err := multiWriter.CreateFormFile("file", "file")
        if err != nil {
            formWriter.CloseWithError(err)
            return
        }

        _, err = io.Copy(partWriter, bodyReader)
        if err != nil {
            formWriter.CloseWithError(err)
            return
        }

        multiWriter.Close()
        formWriter.Close()
    }()

    url := "http://example.com/upload"
    req, _ := http.NewRequest(http.MethodPost, url, formReader)
    fmt.Println(multiWriter.FormDataContentType())
    req.Header.Set("Content-Type", multiWriter.FormDataContentType())
    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        fmt.Println("htttp do error")
        return
    }
    fmt.Println(resp.StatusCode)
    respBody, err := ioutil.ReadAll(resp.Body)
    if err != nil {
        fmt.Println("read error")
        return
    }
    fmt.Println("Result:", string(respBody))
}

another followup question please. If the line`size, err = io.Copy(partWriter, bodyReader)` is made, how can i get size in the main function ? — JR Wink, Oct 11 '21 at 05:41
you could follow this post https://stackoverflow.com/questions/20945069/catching-return-values-from-goroutines — emptyhua, Oct 11 '21 at 07:18

Charlie Tumahai · Answer 2 · 2021-10-11T04:17:29.040

Fix by setting the command's standard output to the part writer.

func main() {
    bodyReader, bodyWriter := io.Pipe()
    formWriter := multipart.NewWriter(bodyWriter)
    go func() (err error) {
        defer func() {
            bodyWriter.CloseWithError(err)
        }()
    
        partWriter, err := formWriter.CreateFormFile("file", "file")
        if err != nil {
            return err
        }
    
        cmd := exec.Command("convert", "test.png", "-resize" "30x30", "png:-")
        cmd.Stdout = partWriter
    
        if err := cmd.Run(); err != nil {
            return err
        }
    
        formWriter.Close()
        return nil
    }()

    ...

}

Run an example on the playground.

Bonus improvement: Use CloseWithError to propagate errors from the goroutine to the main function.

Using io.Pipe to get output from exec.command to make http post

2 Answers2