read on a pipe blocks until program running at end of pipe terminates - Windows

Question

I have a sample program that outputs a line of text every second. In the test program below, this program writes some text to stdout then waits 1 second and repeats 20 times.

I have another program which uses popen (_popen on Windows) to open a pipe for reading from the program. I then use fgets to read data. The problem I have is that the fgets blocks until the program terminates. Then I get all the output, all 20 lines, in one go. I want to get the output a line at a time, then ok for fgets to block until next line ready. The reason is I plan to use this on a program that will be constantly running, outputting text, e.g. like the use of tail.

If I run this code example on a program that outputs some text all in one go and exits then it works fine.

Why does fgets block? The test program does print some text immediately, so why doesn't fgets read this first line of text immediately?

Here is the code:

#include <stdio.h>
#include <windows.h>

void execute(const char* cmd) {
    char buffer[128] = { 0 };
    FILE* pipe = _popen(cmd, "r");

    if (!pipe) {
        printf("popen() failed!\n");
        return;
    }

    while (!feof(pipe)) {
        if (fgets(buffer, 128, pipe) != nullptr)
            printf("%s", buffer);
    }

    int rc = _pclose(pipe);

    if (rc != EXIT_SUCCESS) { // return code not 0
        printf("pclose exit failure: %d\n", rc);
    }
}


int main(int argc, char* argv[]) {
    if (argc != 2) {
        printf("Usage: pipe_test.exe <program>\n");
        exit(1);
    }

    execute(argv[1]);
}

The program run, helloworld.exe:

#include <stdio.h>
#include <windows.h>

int main() {

    for (int i = 0; i < 20; i++) {
        printf("Hello World %d\n", i);
        Sleep(1000);
    }
}

Writing to a pipe is buffered by default. Either turn off buffering with `setvbuf()` or call `fflush(stdout)` after each `printf()`. — Barmar, Jan 20 '20 at 18:00
@Barmar I tried setvbuf(pipe, NULL, _IONBF, 128); after creating the pipe but had no effect — Angus Comber, Jan 20 '20 at 18:11
@Barmar in the called program flush(stdout) means get immediate output. Not as convenient because I would prefer to work with unmodified called programs, but gives me a workaround. I thought \n in the printf in the called program is same as flush? — Angus Comber, Jan 20 '20 at 18:15
@AngusComber printf will flush on newline but only when going to real stdout device. If it is redirected then it is buffered to improve throughput. — Gem Taylor, Jan 20 '20 at 18:27
https://stackoverflow.com/questions/5431941/why-is-while-feof-file-always-wrong — William Pursell, Jan 20 '20 at 19:01
@AngusComber `stdout` is line-buffered when connected to a terminal, fully-buffered when connected to anything else. If you can't modify the child program, the workaround is to use the `unbuffer` utility to run it; this comes with the Expect tool. — Barmar, Jan 20 '20 at 19:43

Marco Bonelli · Accepted Answer · 2020-01-20T18:51:25.820

Why does fgets block?

Because it's waiting for the children to output something.

The test program does print some text immediately, so why doesn't fgets read this first line of text immediately?

It actually does not print text immediately. The problem here, as @Barmar notices, is that writing to a pipe is buffered (and not line buffered) by the C standard library implementation. This buffering happens in your child program (helloworld), not in your parent program (pipe_test).

From your parent program, you have no control over what the children spawned through popen() will do, therefore if the child output is buffered like in this case, the only thing you can do (without modifying the child's code) is to wait until the buffer is flushed to the pipe.

In order to get the output sooner, you would have to modify the children's code to manually call fflush() or use setvbuf() to disable buffering:

int main() {
    setvbuf(stdout, NULL, _IONBF, 0); // Disable buffering on stdout.

    for (int i = 0; i < 20; i++) {
        printf("Hello World %d\n", i);
        Sleep(1000);
    }
}

There's really not much else you can do.

read on a pipe blocks until program running at end of pipe terminates - Windows

1 Answers1