ToUnicode doesn't return correct characters

Question

I tried to call the ToUnicode inside a low level keyboard hook and print the character(s) it returned. However, it seems that the function doesn't take into account whether special keys, such as shift or caps lock were pressed, so the output is the same as from MapVirtualKey function with current key's virtual code passed as parameter.

For example (pressed keys => characters returned by ToUnicode):

abcd => abcd (correct)
[caps lock]abcd => abcd (wrong: should be ABCD)
ab[holding shift]cd => abcd (wrong: should be abCD)

How I call the function (inside the hook procedure):

    KBDLLHOOKSTRUCT* pressedKeyInformation = (KBDLLHOOKSTRUCT*)lParam;

    BYTE keysStates[256]; // 256 bo tyle virtualnych klawiszy wpisze GetKeyboardState

    if(!GetKeyboardState(keysStates))
        //error
    else
    {
        WCHAR charactersPressed[8] = {};

        int charactersCopiedAmount = ToUnicode(pressedKeyInformation->vkCode, pressedKeyInformation->scanCode, keysStates, charactersPressed, 8, 0);

        //std::wcout << ...
    }

Later I noticed that calling GetKeyState with any virtual key code passed as parameter (e.g. VK_RETURN, VK_SHIFT) before ToUnicode causes it to return the correct character, e.g.:

abcd => abcd (correct)
[caps lock]abcd => ABCD (correct)
ab[holding shift]cd => abCD (correct)

It also returns properly keyboard locale dependent keys pressed with AltGr then, e.g. [AltGr]a => ą.

The above example isn't entirely correct, since there appears another problem - if e.g. caps lock was pressed, the next character still depends on its previous state, only the latter characters are affected, e.g.:

abcd => abcd (correct)
(caps lock is off)[caps lock]abcd => aBCD (wrong: should be ABCD)
(caps lock is off)ab[caps lock]cd => abcD (wrong: should be abCD)

Have you any idea why the GetKeyState(<whatever>) fixes one of the problems and what's the cause of the latter caps lock (and other special keys) problem?

Call both `GetKeyState(VK_SHIFT)` and `GetKeyState(VK_CAPITAL)` before calling `GetKeyboardState`. Your [MCVE](https://stackoverflow.com/help/mcve) is not clear. Are you responding to `WM_KEYDOWN`? — Barmak Shemirani, Dec 10 '18 at 04:03
@Barmak Shemirani Yes, I'm responding to `WM_KEYDOWN`. I'll change the code once i'll be home, so in a couple of hours. — Jason, Dec 10 '18 at 05:59
@BarmakShemirani It works perfectly now, thanks. But what do these calls actually do? I believe `GetKeyState(key)` updates `key`'s state internally, am I right? I can't find any clue on msdn. You should post your comment as answer. — Jason, Dec 10 '18 at 15:13

Barmak Shemirani · Accepted Answer · 2018-12-10T23:56:31.187

1

Partial answer:

Windows documentation suggests GetKeyboardState and GetKeyState return similar result for the correspond keys, and this is true when these functions are used in a Windows message loop, where keyboard messages are properly translated.

In this case however, we have a hook function, GetKeyboardState doesn't properly fill the keyboard. Calling GetKeyState first, will change the keyboard state, the subsequent call to GetKeyboardState will work as expected. I don't know why!

Other oddities, GetKeyState returns SHORT value, while GetKeyboardState fills BYTE array. But this shouldn't make a difference since we are only interested in high and low bits.

HHOOK hook;
LRESULT CALLBACK hook_procedure(int code, WPARAM wparam, LPARAM lparam)
{
    if(code == HC_ACTION)
    {
        if(wparam == WM_KEYDOWN)
        {
            KBDLLHOOKSTRUCT *kb = (KBDLLHOOKSTRUCT*)lparam;
            BYTE state[256] = { 0 };
            wchar_t str[10] = { 0 };
            GetKeyState(VK_SHIFT);
            GetKeyState(VK_MENU);
            GetKeyboardState(state);
            if (ToUnicode(kb->vkCode, kb->scanCode, 
                state, str, sizeof(str)/sizeof(*str) - 1, 0) > 0)
            {
                if(kb->vkCode == VK_RETURN) std::wcout << "\r\n";
                else std::wcout << str;
            }
        }
    }
    return CallNextHookEx(hook, code, wparam, lparam);
}

edited Dec 10 '18 at 23:56

answered Dec 10 '18 at 20:17

Barmak Shemirani

30,904
6
40
77

Thanks! One note for displaying the characters returned by `ToUnicode` - according to msdn: `However, the buffer may contain more characters than the return value specifies. When this happens, any extra characters are invalid and should be ignored.`, so it'd be nice if you'd display only `N` characters, where `N` is a value returned by `ToUnicode`. – Jason Dec 10 '18 at 21:07
@Jason, yes that makes sense. Does this work with `AltGr` as well? – Barmak Shemirani Dec 10 '18 at 22:28
If you meant whether dead keys produce a single character, the answer is yes, but only if the dead key could be combined with another pressed key - according to msdn: `The most common cause for this [i.e. ToUnicode buffer contains >= 2 characters] is that a dead-key character (accent or diacritic) stored in the keyboard layout could not be combined with the specified virtual key to form a single character.`By the way, if we have to follow the rules, you should `return CallNextHookEx` if `code < 0`, before intercepting any keys, as specified in documentation of the hook proc. – Jason Dec 10 '18 at 23:14
Also, if there were copied 8 characters to the str buffer, then `str[8] = 0` would cause undefined behaviour. However, still: `this buffer may be returned without being null-terminated even though the variable name suggests that it is null-terminated.`. – Jason Dec 10 '18 at 23:25
Okay, I added an extra index to the buffer. – Barmak Shemirani Dec 10 '18 at 23:57

ToUnicode doesn't return correct characters

1 Answers1

Linked