SetWindowsHookEx(WH_JOURNALRECORD, ..) sometimes hang the system

Question

I'm trying upgrade an old application to work on Windows 7 and I'm having problems with the "Macro recording" functionality, which is using Journal Hooks. I have followed every step necessary to make this work on Windows 7, that is, setting the uiAccess=true, signing the exe and running it from the Program Files directory.

It usually works, but sometimes, for no apparent reason, it seems like the SetWindowsHookEx function is waiting for something and hang the whole system in a weird way : no input is send to any application. The only way I can get out of this hang is by doing ctrl-alt-del, which force uninstall the hook.

I've replicated the problem with a simple application. Using the default generated Win32 template of Visual Studio, I've modified the About dialog callback to register and unregister the hook :

// Message handler for about box.
INT_PTR CALLBACK About(HWND hDlg, UINT message, WPARAM wParam, LPARAM lParam)
{
    UNREFERENCED_PARAMETER(lParam);
    switch (message)
    {
    case WM_INITDIALOG:
    {
        // added this
        recordHook = ::SetWindowsHookEx(WH_JOURNALRECORD, JournalRecordCallback, hInst, 0);
        return (INT_PTR)TRUE;
    }
    case WM_COMMAND:
        if (LOWORD(wParam) == IDOK || LOWORD(wParam) == IDCANCEL)
        {
            // and this
            UnhookWindowsHookEx(recordHook);
            EndDialog(hDlg, LOWORD(wParam));
            return (INT_PTR)TRUE;
        }
        break;
    }
    return (INT_PTR)FALSE;
}

My hook callback function is simply

LRESULT CALLBACK JournalRecordCallback(int nCode, WPARAM wParam, LPARAM lParam)
{
    return CallNextHookEx(NULL, nCode, wParam, lParam);
}

And, as with the larger application, sometimes, the SetWindowsHookEx call hang.

Does anybody has experienced this? I thought that maybe installing the hook inside the MessageLoop was causing the hang and I tried to move it to another thread but still got the hang.

Am I doing something wrong? Thanks

Answer 1

JournalRecordProc callback function:

An application that has installed a JournalRecordProc hook procedure should watch for the VK_CANCEL virtual key code (which is implemented as the CTRL+BREAK key combination on most keyboards). This virtual key code should be interpreted by the application as a signal that the user wishes to stop journal recording. The application should respond by ending the recording sequence and removing the JournalRecordProc hook procedure. Removal is important. It prevents a journaling application from locking up the system by hanging inside a hook procedure.