Getting a console window out of "select mode" to prevent a crash

Question

In my C# application, I am trying to prevent a crash in my application, basically, I am using a Console window as a log display window. I already solved the "Close Button" issue by disabling the close window, and I show/hide the menu with Show and Hide calls, all of that is working just fine.

My final hurtle is if Text Selection is active and the window attempting to be hidden.

I either need to:

A. Kick the window out of select mode. (not sure how I would do this, since selection pauses all output.)

B. Disable the "Edit" menu in the same way I disabled the Close menu, in a hope that it would also disable the mouse selection, but I have yet to find any way to remove the "Edit" menu, and I am not even sure that would prevent mouse selection.

C. This seems the most obvious, disable mouse selection, and this is the one I have currently in my code, but it's not working, so I am not sure what I am missing.

The Code in Question:

[DllImport("user32.dll")]
private static extern bool ShowWindow(IntPtr hWnd, int nCmdShow);
 ....
[DllImport("Kernel32.dll", ExactSpelling = true)]
private static extern int SetConsoleMode(IntPtr hWnd, int wFlag);
 ....
public static void HideConsoleLog () {
    if (handle != IntPtr.Zero) {
        SetConsoleMode(handle, 0x0080);  // ENABLE_EXTENDED_FLAGS 0x0080
        ShowWindow(handle, (int)WinCntrlOpt.SW_HIDE);
        LogVisible = false;
    }
}

According to the Documentation:

ENABLE_EXTENDED_FLAGS 0x0080 Required to enable or disable extended flags. See ENABLE_INSERT_MODE and ENABLE_QUICK_EDIT_MODE.

And...

ENABLE_QUICK_EDIT_MODE 0x0040 This flag enables the user to use the mouse to select and edit text.

To enable this mode, use ENABLE_QUICK_EDIT_MODE | ENABLE_EXTENDED_FLAGS. To disable this mode, use ENABLE_EXTENDED_FLAGS without this flag.

So, it seems I am doing this correctly, and it does run, but it does not prevent mouse selection. I do know that not to long ago, Microsoft tinkered with Consoles, and I don't know if that's broke this or not.

There is a similar question on stack exchange, but that nether gives a working solution, and is in-fact the reverse to what I am trying to do and in C or C++. I can interpolate, but it's basically what I am already trying here.

It seems to be the same suggestion, use ENABLE_EXTENDED_FLAGS without any other flags to disable the option, and that's what I have. But it doesn't work.

So, I am either scratching at the wrong solution or am not doing something else correctly.

The effect is, while the console window is open, you can use the mouse to select text which automatically puts the console in that "select mode" that Microsoft's tinkering added. That's not a problem, but it's when it's left IN that mode, and you select "Show Log" (a check-marked toggle menu option in the main application window) The console window does indeed hide, but then the main window locks up and the application crashes. (my guess is because the window is blocked by that select mode.) It doesn't crash when the 'select mode' is not active.

I would also like to try this from the same documentation:

ENABLE_MOUSE_INPUT 0x0010 If the mouse pointer is within the borders of the console window and the window has the keyboard focus, mouse events generated by mouse movement and button presses are placed in the input buffer. These events are discarded by ReadFile or ReadConsole, even when this mode is enabled.

Problem is, it says:

When a console is created, all input modes except ENABLE_WINDOW_INPUT are enabled by default.

So, how do I disable it?

I am fine with any solution that either kicks it out of select mode, or prevents it to begin with, and I have here what I have tried.

Instead of disabling console features, why not creating a winform application with only the desired feature ? — Cyril Durand, Feb 15 '19 at 15:29

score 4 · Accepted Answer · answered Feb 12 '19 at 11:26

When the "Show Log" menu option is selected, the window is going to minimize. Before the window minimizes, you have to check whether the user has selected any input to prevent the crash. You can check whether the user is selecting anything by using GetConsoleSelectionInfo.

The CONSOLE_SELECTION_INFO out parameter should be equal to 0x00, and if it's not, you need to process the selection. As the documentation from GetConsoleMode/SetConsoleMode shows, you either need to call ReadFile or ReadConsole to discard the selection event that might be going on.

When you've implemented this, it shouldn't crash anymore.

This really helped! Thank you! I put in [DllImport("Kernel32.dll", ExactSpelling = true)] [return: MarshalAs(UnmanagedType.Bool)] private static extern bool GetConsoleSelectionInfo(out IntPtr lpConsoleSelectionInfo); and checked the result pointer for IntPtr.Zero and it worked as you described. — Quade2002, Feb 12 '19 at 19:33

Getting a console window out of "select mode" to prevent a crash

1 Answers1