Why does reopening stdout to a newly-allocated console occasionally fail?

Question

Consider this program:

#include <windows.h>
#include <stdio.h>

#pragma comment(linker, "/SUBSYSTEM:CONSOLE")

int main(void) {
    if (!FreeConsole()) {
        MessageBoxA(NULL, "FreeConsole failed", NULL, MB_ICONHAND);
        return 1;
    }
    if (!AllocConsole()) {
        MessageBoxA(NULL, "AllocConsole failed", NULL, MB_ICONHAND);
        return 1;
    }
    FILE* newstdout;
    if (freopen_s(&newstdout, "CONOUT$", "w", stdout)) {
        MessageBoxA(NULL, "freopen_s failed", NULL, MB_ICONHAND);
        return 1;
    }
    if (newstdout != stdout) {
        MessageBoxA(NULL, "freopen_s did something wonky", NULL, MB_ICONHAND);
        return 1;
    }
    puts("1");
    MessageBoxA(NULL, "Look for 1 then press OK", NULL, 0);
    return 0;
}

Steps to reproduce: create a new empty C/C++ project in Visual Studio 2019, go to Project Properties -> Configuration Properties -> Linker -> System and change SubSystem to Not Set, then add the above code to the project in a new file called Source.c.

Most of the time I run it, it opens a new console with the number 1 in it, like it's supposed to, but sometimes it fails. If I used "Start Without Debugging" (Ctrl+F5) to run it, it fails about 1 run out of 30, with my "freopen_s failed" message. If I used "Start Debugging" (F5) to run it, it also fails about 1 run out of 10, with _NtClose throwing an exception that says "An invalid handle was specified", with a stack trace pointing at my call to freopen_s.

Why does this intermittent failure happen? Am I breaking one of the rules of the Windows API, either in the C code or in the project configuration?

Things I tried so far that didn't change the behavior:

Removing the line with #pragma
Using CON in place of CONOUT$
freopen_sing stdout to NUL before calling FreeConsole

@Joshua Because that's what [this](https://stackoverflow.com/a/9047596/7509065), [this](https://stackoverflow.com/a/15547699/7509065), and [this](https://stackoverflow.com/a/67461918/7509065) used. Although now that you said that, I did just try switching to `CON` and it still did the exact same thing. — Joseph Sible-Reinstate Monica, Jun 07 '22 at 05:12
Why would you ever want FreeConsole followed by AllocConsole? — Anders, Jun 07 '22 at 16:16
@Anders: To detach from the calling process's console and have your own. I've done this in Win32 code (no C standard library) — Joshua, Jun 07 '22 at 16:35
@Joshua That is what the `start` command is for :). `FreeConsole` assumes that you are done with consoles from then on. — Anders, Jun 07 '22 at 17:24
@Anders [The MSDN documentation for FreeConsole](https://learn.microsoft.com/en-us/windows/console/freeconsole) says "After a process calls FreeConsole, it can call the AllocConsole function to create a new console or AttachConsole to attach to another console." — Joseph Sible-Reinstate Monica, Jun 07 '22 at 17:58

score 1 · Answer 1 · answered Jun 07 '22 at 04:45

This is messed up, it's not your fault. FreeConsole() proceeds to destroy the old console and take down the handle with it. This invalidates stdout asynchronously.

You really want to do something like this. Note that you really do need to do stdin and stderr as well as stdout or bad things can happen later.

    // Change the handle of stdout so that we can destroy the old one.
    FILE* newstdout;
    if (freopen_s(&newstdout, "NUL", "w", stdout)) {
        MessageBoxA(NULL, "freopen_s failed", NULL, MB_ICONHAND);
        return 1;
    }
    if (!FreeConsole()) {
        MessageBoxA(NULL, "FreeConsole failed", NULL, MB_ICONHAND);
        return 1;
    }
    if (!AllocConsole()) {
        MessageBoxA(NULL, "AllocConsole failed", NULL, MB_ICONHAND);
        return 1;
    }
    // Now we can point stdout at it.
    if (freopen_s(&newstdout, "CONOUT$", "w", stdout)) {
        MessageBoxA(NULL, "freopen_s failed", NULL, MB_ICONHAND);
        return 1;
    }
    if (newstdout != stdout) {
        MessageBoxA(NULL, "freopen_s did something wonky", NULL, MB_ICONHAND);
        return 1;
    }

Unfortunately, even when I do switch `stdout` to `NUL` first like that, the same thing still happens. — Joseph Sible-Reinstate Monica, Jun 07 '22 at 05:13

Why does reopening stdout to a newly-allocated console occasionally fail?

1 Answers1