How does cmd.exe display stdout even though the process doesn't flush?

Question

I used this MSDN code to hook stdout and stderr for a child process, (except that I'm calling PeekNamedPipe first so that I don't block) and all is well except that for stdout if the process doesn't flush then I don't see the output of the process immediately. Stderr seems to be automatically flushed whenever something is sent to it, but stdout buffers the results.

Suddenly I'm amazed that the cmd.exe shell process can print results immediately when you step through code in Visual Studio. I would like to do so too. How does it do it?

very simply. not use c/c++ library for output. use winapi - `WriteFile` or `WriteConsoleW` — RbMm, Oct 28 '19 at 06:48
@Eelke, C standard I/O defaults to full buffering when writing to stdout if it's a pipe or disk file. However, cmd.exe does not use C standard I/O -- or even C low I/O (e.g. `_write`), but rather uses the Windows API directly. If it's writing to a console, CMD uses `WriteConsoleW`. Otherwise it writes encoded bytes to stdout via `WriteFile`. If CMD's `/u` option is specified, it writes native UTF-16LE, else it encodes via `WideCharToMultiByte`. For the latter, with `/a` it uses the system ANSI codepage, else it uses the current console input codepage. — Eryk Sun, Oct 28 '19 at 07:09
For blocking issue you can check [this](https://stackoverflow.com/questions/58294219/createprocess-stdout-with-createnamedpipe-overlapped) similar question. — Rita Han, Oct 29 '19 at 03:12

score 0 · Answer 1 · answered Oct 28 '19 at 06:13

0

It's not the cmd.exe rather the implementation of the stdio. When printing to stdout it holds the information in a buffer and flushes it periodically (or before shutdown) by a different thread. This behavior is on purpose and used to improve performance. Otherwise, printing will be blocking.

Here is how to make it autoflush after each print command: C autoflush

answered Oct 28 '19 at 06:13

Roee Gavirel

18,955
12
67
94

1

cmd not use stdio at all – RbMm Oct 28 '19 at 06:46
true, but the MSDN code in the link he shared (the one he is running from the cmd) does. – Roee Gavirel Oct 28 '19 at 06:48
no. even code example under link not use stdio but call `WriteFile` and `ReadFile`. cmd also not use stdio – RbMm Oct 28 '19 at 06:50
What do you think the header `#include ` and all the `printf()` in the code does, then? – Roee Gavirel Oct 28 '19 at 06:52
in example used `ReadFile` and `WriteFile` than and question was not about this example but about cmd.exe. your answer is wrong. cmd exe not use stdio and no any autoflush – RbMm Oct 28 '19 at 06:55
1

Just to be clear for the purposes of this question, please note that cmd.exe bypasses C standard (e.g. `fwrite`) and low (e.g. `_write`) I/O to instead use the Windows API directly. – Eryk Sun Oct 28 '19 at 06:57
The question is, why when using cmd.exe to run the code in the MSDN code example then `except that for stdout if the process doesn't flush then I don't see the output of the process immediately`. – Roee Gavirel Oct 28 '19 at 06:58
@RoeeGavirel, Jorge wants to know what cmd.exe is doing to send data directly to the pipe without a `flush`, and the answer is simply that cmd.exe does not use standard I/O at all in this case -- no `fwrite`, no `printf`. It writes directly to the pipe via `WriteFile`. – Eryk Sun Oct 28 '19 at 07:14
The question is: When debugging an app in Visual Studio it doesn't require a ``fflush(stdout)`` every time I printf, but in a debugged child process I create it does. What I can do to eliminate the buffering in the child process that I'm debugging? I don't control whether the child process uses std or winapi. This answer would be appropriate if I can invoke setvbuf in a child process, but I can't figure out how to do so. If I can remove buffering from a child process, that's the answer. – Jorge Rodriguez Oct 28 '19 at 19:11
1

@JorgeRodriguez, in Linux there's a `stdbuf` program for this, but it relies on a couple features that are not generically supported in Windows: (1) the `LD_PRELOAD` environment variable that allows injecting shared libraries into a process before the normally loaded libraries and (2) that there's typically only one well-known C runtime (libc.so.6) loaded in the global namespace, thus making it reliable to call `setvbuf` before the application's `main` entry point is called. Windows has no such loader hook, no global namespace for loaded modules, and commonly has multiple CRTs in a process. – Eryk Sun Oct 28 '19 at 20:43
@JorgeRodriguez, there's also the Expect pty approach. If there's not much output, you can create a console screen buffer for the child, with the maximum 9,999 lines. Then read it at the end via `ReadConsoleOutputW`. Or actively poll/scrape its contents, but that's not very reliable. Windows 10 adds pseudoconsole support. This allows you to start a child process attached to a custom console (conhost.exe), so the child has console I/O instead of pipes. On the backend, this console and your process communicate via UTF-8 virtual-terminal streams over a pair of pipes (input & output). – Eryk Sun Oct 28 '19 at 20:57
The Windows debug app does insert a fake breakpoint before it invokes main which allows me to do arbitrary operations on the guest process, so if I can figure out what to call then I know where to put it, but there’s no setvbuf for external processes equivalent. My stopgap is that I can write some code to the guest process that calls this, set RIP to that code, and then return it when done... super hacky. I’m hoping for some native support. – Jorge Rodriguez Oct 29 '19 at 06:55

score 0 · Answer 2 · answered Oct 28 '19 at 07:05

0

cmd.exe use direct winapi calls WriteFile or WriteConsoleW for print to stdout. so if you want to do so too - you can also use this winapi.

when stdout is file:

when stdout is console:

answered Oct 28 '19 at 07:05

RbMm

31,280
3
35
56

score 0 · Answer 3 · answered Oct 29 '19 at 03:08

We can't find out how the Visual Studio implement this logic.

To solve your issue, prevent the parent process blocking and print the output from the child process, you can check the read size returned from ReadFile in your parent process to decide to read continue or stop if you can't control the child process.

for (;;)
{
    bSuccess = ReadFile(g_hChildStd_OUT_Rd, chBuf, BUFSIZE, &dwRead, NULL);
    if (!bSuccess || dwRead == 0) break;

    bSuccess = WriteFile(hParentStdOut, chBuf,
        dwRead, &dwWritten, NULL);
    if (!bSuccess) break;

    // Check if there is more data to read, if not break.
    if (dwRead < BUFSIZE)
        break;
}

How does cmd.exe display stdout even though the process doesn't flush?

3 Answers3