Block ESC and Enter keys in modeless dialog box (Win32, non-MFC)

Question

There're some articles written on this subject, but none of them worked in my case. I'm writing the following using Win32 (no MFC). The goal is to prevent ESC or ENTER keys from closing the modeless dialog box.

Here's the dialog template:

IDD_DIALOG_1 DIALOGEX 0, 0, 345, 179
STYLE DS_SETFONT | DS_FIXEDSYS | WS_MAXIMIZEBOX | WS_POPUP | WS_CAPTION | WS_SYSMENU | WS_THICKFRAME
CAPTION ""
FONT 8, "MS Shell Dlg", 400, 0, 0x1
BEGIN
    CONTROL         "New Pt",IDC_CHECK_NEW_PT,"Button",BS_AUTOCHECKBOX | BS_PUSHLIKE | WS_TABSTOP,7,3,39,12
    CONTROL         "Lines",IDC_CHECK_LINES,"Button",BS_AUTOCHECKBOX | BS_PUSHLIKE | WS_TABSTOP,54,3,39,12
    CONTROL         "Curves",IDC_CHECK_CURVES,"Button",BS_AUTOCHECKBOX | BS_PUSHLIKE | WS_TABSTOP,94,3,39,12
    CONTROL         "Ellipses",IDC_CHECK_ELLIPSE,"Button",BS_AUTOCHECKBOX | BS_PUSHLIKE | WS_TABSTOP,134,3,39,12
    CONTROL         "Circles",IDC_CHECK_CIRCLE,"Button",BS_AUTOCHECKBOX | BS_PUSHLIKE | WS_TABSTOP,174,3,39,12
    LTEXT           "Pen Size:",IDC_STATIC,242,7,30,8
    EDITTEXT        IDC_EDIT_PEN_SIZE,275,3,40,14,ES_CENTER | ES_AUTOHSCROLL | ES_NUMBER
    CONTROL         "",IDC_SPIN_PEN_SIZE,"msctls_updown32",UDS_SETBUDDYINT | UDS_ALIGNRIGHT | UDS_AUTOBUDDY | UDS_ARROWKEYS,316,3,11,14
    EDITTEXT        IDC_EDIT_SRC,7,19,331,106,ES_MULTILINE | ES_AUTOVSCROLL | ES_AUTOHSCROLL | ES_WANTRETURN | WS_VSCROLL | WS_HSCROLL
END

To trap those two keys, I change the message loop to this:

MSG msg;

// Main message loop:
for(int nR;;)
{
    nR = ::GetMessage(&msg, nullptr, 0, 0);
    if(!nR)
    {
        break;
    }
    else if(nR == -1)
    {
        //Error
        ASSERT(NULL);
        break;
    }

    if(ghActiveModelessDlg)
    {
        BOOL bProcessAsDlgMsg = TRUE;

        if(msg.message == WM_KEYDOWN ||
            msg.message == WM_KEYUP)
        {
            //Try to trap ESC & Enter keys
            if(msg.wParam == VK_ESCAPE)
            {
                //Do not process
                bProcessAsDlgMsg = FALSE;
            }
            else if(msg.wParam == VK_RETURN)
                goto lbl_check_enter;
        }
        else if(msg.message == WM_CHAR)
        {
            //Try to trap ESC & Enter key
            if(msg.wParam == 27)
            {
                //ESC - Do not process
                bProcessAsDlgMsg = FALSE;
            }
            else if(msg.wParam == '\r')
            {
lbl_check_enter:
                //See what window is it
                WCHAR buffClass[256];
                if(::GetClassName(msg.hwnd, buffClass, _countof(buffClass)) &&
                    lstrcmpi(buffClass, L"edit") == 0 &&
                    (::GetWindowLongPtr(msg.hwnd, GWL_STYLE) & ES_WANTRETURN))
                {
                    //This is edit ctrl that can handle its own Enter keystroke
                }
                else
                {
                    //Do not process
                    bProcessAsDlgMsg = FALSE;
                }
            }
        }

        if(bProcessAsDlgMsg)
        {
            if(::IsDialogMessage(ghActiveModelessDlg, &msg))
            {
                continue;
            }
        }
    }

    if (!TranslateAccelerator(msg.hwnd, hAccelTable, &msg))
    {
        TranslateMessage(&msg);
        DispatchMessage(&msg);
    }
}

And ghActiveModelessDlg is set from within DlgProc for the modeless dialog as such:

INT_PTR CALLBACK DlgProc(HWND hDlg, UINT message, WPARAM wParam, LPARAM lParam)
{
    switch(hDlg)
    {
        //...

        case WM_ACTIVATE:
        {
            //Needed to ensure that keyboard shortcuts are properly processed in the message loop
            ghActiveModelessDlg = wParam != WA_INACTIVE ? hDlg : NULL;
        }
        break;
    }

    return 0;
}

This works ... in most cases. Except this one.

Here's the sequence. Put the focus into the multi-line edit box, then hit any letter/number key and then ESC:

It will then close the dialog.

Any idea how can it pass my override code above?

PS. Interesting observations.

1) If I just hit ESC first, my code traps it. It's only when I hit some other key and then ESC it fails.

2) If I comment out the line that calls IsDialogMessage (and a subsequent continue) it stops accepting ESC. So my guess is that it's not the edit control that does this.

Answer 1

if we want let close dialog only by clicking close X button in system menu (or by ALT+F4) and disable close by ESC and ENTER key - all what we need - call DestroyWindow when process (WM_SYSCOMMAND, SC_CLOSE) and do nothing on (WM_COMMAND, IDCANCEL, IDOK). we not need special message loop or subcluss any controls. and not have buttons with IDOK/ IDCANCEL id in dialog

INT_PTR DialogProc(HWND hwndDlg, UINT uMsg, WPARAM wParam, LPARAM lParam)
{

    switch (uMsg)
    {
    case WM_SYSCOMMAND:
        if ((wParam & 0xfff0) == SC_CLOSE) DestroyWindow(hwndDlg);
        break;
    case WM_COMMAND:
        switch (wParam)
        {
        case MAKEWPARAM(IDOK, BN_CLICKED):
        case MAKEWPARAM(IDCANCEL, BN_CLICKED):
            // ignore this
            break;
        ....
        }
    }
    ....
}

Answer 2

IsDialogMessage translates ESC key into WM_COMMAND IDCANCEL and ENTER into WM_COMMAND IDOK. To suppress default handling (closing dialog), process them in your dialog procedure:

switch (message)
{
case WM_CLOSE:
    // Handle WM_CLOSE here so it wouldn't generate WM_COMMAND IDCANCEL
    // that would be ignored in WM_COMMAND handler.
    DestroyWindow(hDlg);
    return TRUE;
case WM_COMMAND:
    if ( LOWORD(wParam) == IDCANCEL || LOWORD(wParam) == IDOK )
        // Prevent default handling by original dialog procedure.
        return TRUE;
    break;
// other cases...
}

Answer 3

RbMm has a good solution. So I'll mark it as the answer.

While waiting for a reply I was able to adjust my original message loop and came up with my own solution. So here it is.

Blocking the Enter key is easy. All I needed to do was to define a default button (either in the dialog editor in VS, or by sending the DM_SETDEFID message) and it will handle all the Enter keystrokes.

The gist for blocking ESC keystrokes was not to pass any keyboard messages bearing the ESC keystroke to any common controls (or children of the dialog window.) As @IInspectable quoted in the comments, some of those common controls are quite old and are not implemented up to spec. Moreover, Microsoft usually doesn't fix old UI bugs and simply calls them features.

So I accomplished the fix by the following modification that will re-route (or reflect) all such messages to my DlgProc, which also has the benefit over RbMm's code in that it also allows me to come up with my own processing for the ESC keystrokes.

Also eliminated goto for goto-purists:

MSG msg;

// Main message loop:
for(int nR; nR = ::GetMessage(&msg, nullptr, 0, 0);)
{
    if(nR == -1)
    {
        //Error
        ASSERT(NULL);
        break;
    }

    //Need special processing for modeless dialogs
    if(ghActiveModelessDlg)
    {
        //Try to catch ESC keystrokes
        if(
            ((msg.message == WM_KEYDOWN || msg.message == WM_KEYUP) && msg.wParam == VK_ESCAPE) ||
            (msg.message == WM_CHAR && msg.wParam == 27)
            )
        {
            //Was this message sent to the dialog window?
            if(ghActiveModelessDlg != msg.hwnd)
            {
                //If no, then reflect it to our dialog window
                ::PostMessage(ghActiveModelessDlg, msg.message, msg.wParam, msg.lParam);

                continue;
            }
        }
        else
        {
            //Dialog's special message-processing
            if(::IsDialogMessage(ghActiveModelessDlg, &msg))
            {
                continue;
            }
        }
    }

    //Regular processing
    if (!TranslateAccelerator(msg.hwnd, hAccelTable, &msg))
    {
        TranslateMessage(&msg);
        DispatchMessage(&msg);
    }
}