Disable buffering on redirected stdout Pipe (Win32 API, C++)

Question

I'm spawning a process from Win32 using CreateProcess, setting the hStdOutput and hStdError properties of STARTUPINFO to pipe handles created with CreatePipe. I've got two threads reading the pipes, waiting for data to become available (or the process to complete, at which point it checks that there is no data left before terminating the thread).
As data becomes available, I write the output out to effectively a big textbox.

What's happening is the output is being buffered, so a slow running process just gets chunks of data thrown at the text box, but not "as it happens".

I'm not sure if it's the pipe that's doing the buffering, or something to do with the redirection.

Is there any way to either set the pipe to be unbuffered, or start the process in such a way that the stdout is sent as soon as possible?

I'm testing with a test app that prints lines one second apart

Here is line one
(waits one second)
Here is line two
(waits one second)
... etc

Does it stream smoothly when the process is writing to a console? On linux this is a fairly well-known problem, and the solution is to allocate a pseudo-tty because some programs activate buffering when the output isn't a tty. On Windows it isn't common to check the filetype of stdout, so I wouldn't expect buffering to be different going into a pipe vs to a console. — Ben Voigt, Aug 02 '10 at 06:19
Yes, when it's on the console (ie. cmd.exe), it streams as expected, with the delays and so on. — Dave Brotherstone, Aug 02 '10 at 06:28
How is the process writing to the standard output? I think you might be having C or C++ streams buffering turned on. — wilx, Aug 02 '10 at 08:08
@wilx: Buffering in C and/or C++ runtime libraries would cause console output to be bursty as well. — Ben Voigt, Aug 02 '10 at 13:23
Perhaps the blockiness is introduced on the read end. What buffer size are you using when you read from the pipe? Are you using `ReadFile` or `ReadFileEx` API directly or some wrapper in your framework? What happens if you read with a one-character buffer size? While Oleg's suggestion of `FILE_FLAG_WRITE_THROUGH` isn't going to help a local pipe, using `CreateFile` and `FILE_FLAG_NO_BUFFERING` may. Have you tried that? — Ben Voigt, Aug 02 '10 at 13:29

score 3 · Accepted Answer · answered Aug 02 '10 at 07:43

3

The buffering is probably in the C runtime (printf etc) and there is not much you can do about it (IIRC it does a isatty() check to determine a buffering strategy)

answered Aug 02 '10 at 07:43

Anders

97,548
12
110
164

That would be the answer in POSIX. But Windows has no concept of a tty, nor any `isatty` function. – Ben Voigt Aug 02 '10 at 13:21
1

@Ben Voigt: the MS c runtime still has it: http://msdn.microsoft.com/en-us/library/f4s0ddew%28v=VS.80%29.aspx – Anders Aug 02 '10 at 15:34
shaking head in amazement at the ".NET Framework Equivalent" list, none of which are equivalent in any way – Ben Voigt Aug 02 '10 at 15:51
BTW here's the closest Win32 API equivalent to `isatty`: http://msdn.microsoft.com/en-us/library/aa364960.aspx – Ben Voigt Aug 02 '10 at 15:52
I've ticked this answer, as it looks like the existing code works with other things that show output slowly - eg compilers. I had assumed that the output side was all the same. – Dave Brotherstone Aug 02 '10 at 16:30
I've got half a solution (and was here looking for the other half). I use the Microsoft Detours library, which allows you to hook arbitrary API functions. I wrote a quick DLL which intercepts GetFileType calls and lies to the child process and tells it these pipes are byte devices. isatty returns true now. Unfortunately, putchar still buffers in the child process, don't know why yet. – Mike Elkins Oct 01 '11 at 12:50
@MikeElkins Can you post more information on how you used the "Microsoft Detours library" Is it available for public use? – kuchi Dec 12 '17 at 16:29

score 2 · Answer 2 · answered May 03 '21 at 06:52

In my case the buffering was in the output of the client (as @Anders wrote), which uses normal printf. Maybe this also depends on the implementation of the C runtime (Visual Studio 2019), maybe the runtime detects 'not a console' and enables buffering.

So I disabled the buffering with this call setvbuf(stdout, (char*)NULL, _IONBF, 0); in my client, now I get the output immediately in the pipe in the server.

Just for completeness: Here's how I read the pipe in the server

HANDLE in;
CreatePipe(&in, &startup.hStdOutput, &sa, 0);            // Pipe for stdout of the child

...

char buffer[16384];
DWORD read, total;
while (PeekNamedPipe(in, NULL, 0, &read, &total, NULL))
{
    if (total == 0)
    {
        if (WaitForSingleObject(info.hProcess, 0) == WAIT_OBJECT_0)
        {
            if (PeekNamedPipe(in, NULL, 0, &read, &total, NULL) && total == 0)
                break;
            continue;
        }
        Sleep(10);
        continue;
    }

    if (total > sizeof(buffer))
        total = sizeof(buffer);
    ReadFile(in, buffer, total, &read, NULL);

    ...
}

score 1 · Answer 3 · answered Aug 02 '10 at 06:22

1

There's SetNamedPipeHandleState, but it only controls buffering for remote pipes, not when both ends are on the same computer.

answered Aug 02 '10 at 06:22

Ben Voigt

277,958
43
419
720

Yes, I've seen that, but, as you say, it doesn't change anything when both ends are on the same computer – Dave Brotherstone Aug 02 '10 at 06:31

score 0 · Answer 4 · answered Aug 02 '10 at 11:00

It seems to me you can solve the problem if you set the hStdOutput and hStdError of STARTUPINFO not to pipe handles created with CreatePipe, but instead of that you create a named pipes (with CallNamedPipe function exactly like you used if before also using SECURITY_ATTRIBUTES with bInheritHandle = TRUE, see http://msdn.microsoft.com/en-us/library/aa365782.aspx) and then open there by name with respect of CreateFile using FILE_FLAG_WRITE_THROUGH flag. Like you can read on the MSDN (http://msdn.microsoft.com/en-us/library/aa365592.aspx):

The pipe client can use CreateFile to enable overlapped mode by specifying FILE_FLAG_OVERLAPPED or to enable write-through mode by specifying FILE_FLAG_WRITE_THROUGH.

So just reopen the pipe with respect of CreateFile using FILE_FLAG_WRITE_THROUGH flag and set the handle/handles to hStdOutput and hStdError of STARTUPINFO.

FILE_FLAG_WRITE_THROUGH 0x80000000 Write-through mode is enabled. This mode affects only write operations on byte-type pipes and, then, **only when the client and server processes are on different computers.** If this mode is enabled, functions writing to a named pipe do not return until the data written is transmitted across the network and is in the pipe's buffer on the remote computer. If this mode is not enabled, the system enhances the efficiency of network operations by buffering data until a minimum number of bytes accumulate or until a maximum time elapses. — Ben Voigt, Aug 02 '10 at 13:21

Disable buffering on redirected stdout Pipe (Win32 API, C++)

4 Answers4

Linked