How to capture git clone output with fork() exec()

Question

here is my full code, I use:

    char *args[] = {"git", "clone", "https://github.com/thinker3/youdao.git", "/tmp/youdao", NULL};

    if (execv("/usr/bin/git", args) < 0) {
        perror("error on exec");
        exit(0);
    }

to run cmd in child, and capture output with:

    char buffer[1024];
    close(pipefd[1]);  // close the write end of the pipe in the parent
    while (read(pipefd[0], buffer, sizeof(buffer)) != 0) {
        std::cout << buffer << std::endl;
    }

in super process, I get

Cloning into '/tmp/youdao'...

but I hope get output like terminal, how to do that?

roroco@roroco-Zhaoyang-K49 /tmp $ git clone https://github.com/thinker3/youdao.git
Cloning into 'youdao'...
remote: Counting objects: 434, done.
remote: Total 434 (delta 0), reused 0 (delta 0), pack-reused 434
Receiving objects: 100% (434/434), 221.51 KiB | 140.00 KiB/s, done.
Resolving deltas: 100% (275/275), done.
Checking connectivity... done.

update

of course, I can do it with libgit2, but it just a question: why I can't get git clone full stdout

Answer 1

Apparently there are minor differences depending on the git version, so I will describe the behavior of the two versions I tested with. Sometime between these two versions the behavior changed from the first to the second.

git version 1.8.3

The only part from that output that git writes to stdout is this:

Cloning into 'youdao'...

The rest of it, the actual cloning progress, gets written to stderr and only if stderr is attached to a terminal. To make it always send the progress to stderr, even when it is not attached to a terminal (like in your case), you have to call it with --progress. See man git-clone:

--progress

Progress status is reported on the standard error stream by default when it is attached to a terminal, unless -q is specified. This flag forces progress status even if the standard error stream is not directed to a terminal.

After you add this parameter you will also have to:

• fix the code that reads from the pipe (read() returns the number of bytes read and doesn't null-terminate the buffer);
• parse the progress you'll get on stderr, because it will not be nicely formatted as you see it in the terminal.

---

Later Edit

Judging from the comments, it appears this wasn't explicit enough, so let me try to say it more clearly.

If you type the command like this in a terminal:

$ git clone "https://github.com/thinker3/youdao.git"

you will get this output in the terminal:

Cloning into 'youdao'...
Receiving objects: 100%
... (rest of progress info)

If you type the command like this instead:

$ git clone "https://github.com/thinker3/youdao.git" 1>output.log

you will get this in output.log:

Cloning into 'youdao'...

and this on the terminal:

Receiving objects: 100%
... (rest of progress info)

This means that git writes Cloning into... on the standard output stream (which you have redirected in the file) and the progress information on the standard error stream, which you have left on the terminal.

If instead you type the command like this on the terminal:

$ git clone "https://github.com/thinker3/youdao.git" 2>output.log

you will get this on the terminal:

Cloning into 'youdao'...

because we already established this is going to stdout, and you left stdout on the terminal.

The file will however be empty! The reason it is empty is described by the bold text at the beginning of the post. Specifically, git prints progress information to stderr only if stderr is going to a terminal. In this case stderr is not going to a terminal because you have redirected it to a file, so git doesn't print the progress information at all.

If you want to force git to print this progress information to stderr even if stderr is not attached to a terminal, you have to tell it explicitly, as described in the original answer, by specifying --progress to git clone, like this:

$ git clone --progress "https://github.com/thinker3/youdao.git" 2>output.log

Now, with the extra parameter and stderr redirected to the file, you will find the progress information in the file. It will however not be as nicely formatted as you see it on the terminal. That's why I mentioned in my original answer that you will have to parse and make sense of what you read from that pipe.

git version 2.5.0

The difference to the above description is that now the first line of output, Cloning into 'youdao'..., also goes to stderr and this line (as opposed to the rest of the progress info text) gets written to stderr no matter if stderr is attached to a terminal or not. The rest of the progress info text is sent to stderr the same as before, only if the error stream goes to a terminal.

So now you have this without the extra parameter I mentioned a few times above:

$ git clone "https://github.com/thinker3/youdao.git" 2>output.log
$ cat output1.log 
Cloning into 'youdao'...

and this with that extra parameter:

$ git clone --progress "https://github.com/thinker3/youdao.git" 2>output.log
$ cat output.log 
Cloning into 'youdao'...
remote: Counting objects: 434, done.        
remote: Total 434 (delta 0), reused 0 (delta 0), pack-reused 434        
Receiving objects: 100% (434/434), 221.51 KiB | 135.00 KiB/s, done.
Resolving deltas: 100% (275/275), done.
Checking connectivity... done.

Conclusion

If you want to be able to get the full progress information text through that pipe, you have to call git clone with --progress.

The rest of the points from the original answer remain the same:

• you will have to fix your pipe reading code
• you will have to parse the output you get through the pipe because it will not be the same as what you see on the terminal or in the file.

Answer 2

Also related to this topic, a way to pipe stderr that could be useful is explained with detail here

command 2>&1 >/dev/null | grep 'something'