0

I am new to golang so please review the code and suggest any changes required.

So the problem statement goes below, We have a file whose contents are in binary and are encrypted. The only way to read that contents if by using a custom utility say (named decode_it).. The command just accepts filename like below

decode_it filename.d

Now what I have to do is live monitoring the output of the decode_it utility in GO. I have written the code which is working great but somehow it is not able to process the latest tailed output (it is waiting for some amount of time for reading the last latest chunk before more data comes in ). s.Scan() is the function which is not returning the latest changes in output of that utility. I have another terminal side by side so I know that a line is appended or not. The GO Scan() function only scans when another chunk is appended at the end.

Please help. Suggest any changes required and also if possible you can suggest any other alternative approach for this.

Output of utility is - These are huge and come in seconds

1589261318 493023 8=DECODE|9=59|10=053|34=1991|35=0|49=TEST|52=20200512-05:28:38|56=TEST|57=ADMIN|
1589261368 538427 8=DECODE|9=59|10=054|34=1992|35=0|49=TEST|52=20200512-05:29:28|56=TEST|57=ADMIN|
1589261418 579765 8=DECODE|9=59|10=046|34=1993|35=0|49=TEST|52=20200512-05:30:18|56=TEST|57=ADMIN|
1589261468 627052 8=DECODE|9=59|10=047|34=1994|35=0|49=TEST|52=20200512-05:31:08|56=TEST|57=ADMIN|
1589261518 680570 8=DECODE|9=59|10=053|34=1995|35=0|49=TEST|52=20200512-05:31:58|56=TEST|57=ADMIN|
1589261568 722516 8=DECODE|9=59|10=054|34=1996|35=0|49=TEST|52=20200512-05:32:48|56=TEST|57=ADMIN|
1589261618 766070 8=DECODE|9=59|10=055|34=1997|35=0|49=TEST|52=20200512-05:33:38|56=TEST|57=ADMIN|
1589261668 807964 8=DECODE|9=59|10=056|34=1998|35=0|49=TEST|52=20200512-05:34:28|56=TEST|57=ADMIN|
1589261718 853464 8=DECODE|9=59|10=057|34=1999|35=0|49=TEST|52=20200512-05:35:18|56=TEST|57=ADMIN|
1589261768 898758 8=DECODE|9=59|10=031|34=2000|35=0|49=TEST|52=20200512-05:36:08|56=TEST|57=ADMIN|
1589261818 948236 8=DECODE|9=59|10=037|34=2001|35=0|49=TEST|52=20200512-05:36:58|56=TEST|57=ADMIN|
1589261868 995181 8=DECODE|9=59|10=038|34=2002|35=0|49=TEST|52=20200512-05:37:48|56=TEST|57=ADMIN|
1589261918 36727 8=DECODE|9=59|10=039|34=2003|35=0|49=TEST|52=20200512-05:38:38|56=TEST|57=ADMIN|
1589261968 91253 8=DECODE|9=59|10=040|34=2004|35=0|49=TEST|52=20200512-05:39:28|56=TEST|57=ADMIN|
1589262018 129336 8=DECODE|9=59|10=032|34=2005|35=0|49=TEST|52=20200512-05:40:18|56=TEST|57=ADMIN|
1589262068 173247 8=DECODE|9=59|10=033|34=2006|35=0|49=TEST|52=20200512-05:41:08|56=TEST|57=ADMIN|
1589262118 214993 8=DECODE|9=59|10=039|34=2007|35=0|49=TEST|52=20200512-05:41:58|56=TEST|57=ADMIN|
1589262168 256754 8=DECODE|9=59|10=040|34=2008|35=0|49=TEST|52=20200512-05:42:48|56=TEST|57=ADMIN|
1589262218 299908 8=DECODE|9=59|10=041|34=2009|35=0|49=TEST|52=20200512-05:43:38|56=TEST|57=ADMIN|
1589262268 345560 8=DECODE|9=59|10=033|34=2010|35=0|49=TEST|52=20200512-05:44:28|56=TEST|57=ADMIN|
1589262318 392894 8=DECODE|9=59|10=034|34=2011|35=0|49=TEST|52=20200512-05:45:18|56=TEST|57=ADMIN|
1589262368 439936 8=DECODE|9=59|10=035|34=2012|35=0|49=TEST|52=20200512-05:46:08|56=TEST|57=ADMIN|
1589262418 484959 8=DECODE|9=59|10=041|34=2013|35=0|49=TEST|52=20200512-05:46:58|56=TEST|57=ADMIN|
1589262468 531136 8=DECODE|9=59|10=042|34=2014|35=0|49=TEST|52=20200512-05:47:48|56=TEST|57=ADMIN|
1589262518 577190 8=DECODE|9=59|10=043|34=2015|35=0|49=TEST|52=20200512-05:48:38|56=TEST|57=ADMIN|
1589262568 621673 8=DECODE|9=59|10=044|34=2016|35=0|49=TEST|52=20200512-05:49:28|56=TEST|57=ADMIN|
1589262618 661569 8=DECODE|9=59|10=036|34=2017|35=0|49=TEST|52=20200512-05:50:18|56=TEST|57=ADMIN|
1589262668 704912 8=DECODE|9=59|10=037|34=2018|35=0|49=TEST|52=20200512-05:51:08|56=TEST|57=ADMIN|
1589262718 751844 8=DECODE|9=59|10=043|34=2019|35=0|49=TEST|52=20200512-05:51:58|56=TEST|57=ADMIN|
1589262768 792980 8=DECODE|9=59|10=035|34=2020|35=0|49=TEST|52=20200512-05:52:48|56=TEST|57=ADMIN|
1589262818 840365 8=DECODE|9=59|10=036|34=2021|35=0|49=TEST|52=20200512-05:53:38|56=TEST|57=ADMIN|
1589262868 879185 8=DECODE|9=59|10=037|34=2022|35=0|49=TEST|52=20200512-05:54:28|56=TEST|57=ADMIN|
1589262918 925163 8=DECODE|9=59|10=038|34=2023|35=0|49=TEST|52=20200512-05:55:18|56=TEST|57=ADMIN|
1589262968 961584 8=DECODE|9=59|10=039|34=2024|35=0|49=TEST|52=20200512-05:56:08|56=TEST|57=ADMIN|
1589263018 10120 8=DECODE|9=59|10=045|34=2025|35=0|49=TEST|52=20200512-05:56:58|56=TEST|57=ADMIN|
1589263068 53127 8=DECODE|9=59|10=046|34=2026|35=0|49=TEST|52=20200512-05:57:48|56=TEST|57=ADMIN|
1589263118 92960 8=DECODE|9=59|10=047|34=2027|35=0|49=TEST|52=20200512-05:58:38|56=TEST|57=ADMIN|
1589263168 134768 8=DECODE|9=59|10=048|34=2028|35=0|49=TEST|52=20200512-05:59:28|56=TEST|57=ADMIN|
1589263218 180362 8=DECODE|9=59|10=035|34=2029|35=0|49=TEST|52=20200512-06:00:18|56=TEST|57=ADMIN|
1589263268 220070 8=DECODE|9=59|10=027|34=2030|35=0|49=TEST|52=20200512-06:01:08|56=TEST|57=ADMIN|
1589263318 269426 8=DECODE|9=59|10=033|34=2031|35=0|49=TEST|52=20200512-06:01:58|56=TEST|57=ADMIN|
1589263368 309432 8=DECODE|9=59|10=034|34=2032|35=0|49=TEST|52=20200512-06:02:48|56=TEST|57=ADMIN|
1589263418 356561 8=DECODE|9=59|10=035|34=2033|35=0|49=TEST|52=20200512-06:03:38|56=TEST|57=ADMIN|

Code -

package main

import (
    "bytes"
    "bufio"
    "io"
    "log"
    "os/exec"
    "fmt"
)

// dropCRLR drops a terminal \r from the data.
func dropCRLR(data []byte) []byte {
    if len(data) > 0 && data[len(data)-1] == '\r' {
        return data[0 : len(data)-1]
    }
    return data
}

func newLineSplitFunc(data []byte, atEOF bool) (advance int, token []byte, err error) {
    if atEOF && len(data) == 0 {
        return 0, nil, nil
    }
    if i := bytes.IndexByte(data, '\n'); i >= 0 {
        // We have a full newline-terminated line.
        return i + 1, dropCRLR(data[0:i]), nil
    }
    // If we're at EOF, we have a final, non-terminated line. Return it.
    if atEOF {
        return len(data), dropCRLR(data), nil
    }
    // Request more data.
    // fmt.Println("Returning 0,nil,nil")
    return 0, nil, nil
}

func main() {

    cmd := exec.Command("decode_it", "filename.d", "4", "1")
    var out io.Reader
    {
        stdout, err := cmd.StdoutPipe()
        if err != nil {
            log.Fatal(err)
        }
        stderr, err := cmd.StderrPipe()
        if err != nil {
            log.Fatal(err)
        }
        out = io.MultiReader(stdout, stderr)
    }
    if err := cmd.Start(); err != nil {
        log.Fatal(err)
    }
    // Make a new channel which will be used to ensure we get all output
    done := make(chan struct{})

    go func() {

        // defer cmd.Process.Kill()
        s := bufio.NewScanner(out)
        s.Split(newLineSplitFunc)

        for s.Scan() {
            fmt.Println("---- " + s.Text())
        }
        if s.Err() != nil {
            fmt.Printf("error: %s\n", s.Err())
        }
    }()

    // Wait for all output to be processed
    <-done

    // Wait for the command to finish
    if err := cmd.Wait(); err != nil{
        fmt.Println("Error: " + string(err.Error()))
    }

    // if out closes, cmd closed.
    log.Println("all done")
}

Also, Since scan() is taking a lot of time and goes into a loop from which I am not able to break as well. Please help for that too..

Rush W.
  • 1,321
  • 2
  • 11
  • 19
  • if you propose a different strategy, then you are welcome with new ideas.. I am open to all and would love to work on it. – Rush W. May 12 '20 at 06:10
  • What do you mean by tailed in "latest tailed output". How do you observe the output from another terminal? The output from decode_it goes to your program. How does the terminal get it? – Charlie Tumahai May 12 '20 at 06:57
  • Oh no no.. I meant running just the utility from another terminal.. And side by side the Go run main.go.. to check if there was output in both or not.. – Rush W. May 12 '20 at 06:58
  • Have you confirmed that decode_it does not turn on output buffering if stdout is not a tty? Pipe your "monitoring instance" through cat and see what happens. – Peter May 12 '20 at 08:21
  • Also, your MultiReader is kind of pointless. Stderr won't be read until stdout is closed. Come to think of it, this could also block decode_it eventually, because the stderr buffer (if any) becomes full. – Peter May 12 '20 at 08:24
  • Yes I resolved it using stdbuf command.. It was the buffer problem. – Rush W. May 12 '20 at 09:24
  • 1
    Possible duplicate of https://stackoverflow.com/questions/53831769/catch-buffered-stdout-output-of-exec-command – Charlie Tumahai May 12 '20 at 14:33

2 Answers2

0

try something like this one, i fixed some issues and make it more simple:

package main

import (
    "bufio"
    "fmt"
    "io"
    "log"
    "os/exec"
)

func main() {
    var err error

    // change to your command
    cmd := exec.Command("sh", "test.sh")

    var out io.Reader
    {
        var stdout, stderr io.ReadCloser

        stdout, err = cmd.StdoutPipe()
        if err != nil {
            log.Fatal(err)
        }
        stderr, err = cmd.StderrPipe()
        if err != nil {
            log.Fatal(err)
        }
        out = io.MultiReader(stdout, stderr)
    }

    if err = cmd.Start(); err != nil {
        log.Fatal(err)
    }

    scanner := bufio.NewScanner(out)

    for scanner.Scan() {
        fmt.Println("---- " + scanner.Text())
    }

    if err = scanner.Err(); err != nil {
        fmt.Printf("error: %v\n", err)
    }

    log.Println("all done")
}

test.sh that i used in test:

#!/bin/bash
while [[ 1 = 1 ]]; do
    echo 1
    sleep 1
done

:)

peakle
  • 149
  • 5
0

I tried resolving the above issue using stdbuf -

cmd := exec.Command("stdbuf", "-o0", "-e0", "decode_it", FILEPATH, "4", "1")

Reference link - STDIO Buffering

When programs write to stdout they write with line bufferring. If they are writing to something else, then they use fully buffered mode. golang exec.Command seems to end up using fully buffered mode so using stdbuf forces no buffering.

Rush W.
  • 1,321
  • 2
  • 11
  • 19