ToAscii/ToUnicode in a keyboard hook destroys dead keys

Question

It seems that if you call ToAscii() or ToUnicode() while in a global WH_KEYBOARD_LL hook, and a dead-key is pressed, it will be 'destroyed'.

For example, say you've configured your input language in Windows as Spanish, and you want to type an accented letter á in a program. Normally, you'd press the single-quote key (the dead key), then the letter "a", and then on the screen an accented á would be displayed, as expected.

But this doesn't work if you call ToAscii() or ToUnicode() in a low-level keyboard hook function. It seems that the dead key is destroyed, and so no accented letter á shows up on screen. Removing a call to the above functions resolves the issue... but unfortunately, I need to be able to call those functions.

I Googled for a while, and while a lot of people seemed to have this issue, no good solution was provided.

Any help would be much appreciated!

EDIT: I'm calling ToAscii() to convert the virtual-key code and scan code received in my LowLevelKeyboardProc hook function into the resulting character that will be displayed on screen for the user.

I tried MapVirtualKey(kbHookData->vkCode, 2), but this isn't as "complete" a function as ToAscii(); for example, if you press Shift + 2, you'll get '2', not '@' (or whatever Shift + 2 will produce for the user's keyboard layout/language).

ToAscii() is perfect... until a dead-key is pressed.

EDIT2: Here's the hook function, with irrelevant info removed:

LRESULT CALLBACK keyboard_LL_hook_func(int code, WPARAM wParam, LPARAM lParam) {

    LPKBDLLHOOKSTRUCT kbHookData = (LPKBDLLHOOKSTRUCT)lParam;
    BYTE keyboard_state[256];

    if (code < 0) {
        return CallNextHookEx(keyHook, code, wParam, lParam);
    }

    WORD wCharacter = 0;

    GetKeyboardState(&keyboard_state);
    int ta = ToAscii((UINT)kbHookData->vkCode, kbHookData->scanCode,
                     keyboard_state, &wCharacter, 0);

    /* If ta == -1, a dead-key was pressed. The dead-key will be "destroyed"
     * and you'll no longer be able to create any accented characters. Remove
     * the call to ToAscii() above, and you can then create accented characters. */

    return CallNextHookEx(keyHook, code, wParam, lParam);
}

Please show the code for the call to ToUnicode(). Where do you get the lpKeyState parameter? — jdigital, Dec 27 '09 at 03:00
@jdigital: Well, I'm personally calling ToAscii, but I've read it happens with ToUnicode, too. My call is basically ToAscii(kbHookData->vkCode, kbHookData->scanCode, GetKeyboardState(), &lpchar, 0) in the WH_KEYBOARD_LL hook function. When the ToAscii call returns -1, this means a dead key was pressed... and the dead key is 'destroyed' as I explained above. — 00010000, Dec 27 '09 at 04:14
I deleted my answer, it was completely incorrect based on the assumption that you were referring to c-runtime functions to convert strings rather than Win32 apis to convert scancodes... sorry. — John Knoeller, Dec 27 '09 at 04:31
I would help a lot if you could describe the purpose of doing this, we may find other solutions for your problem. — sorin, Dec 27 '09 at 13:26
@Sorin: See the edit to my original question for more info. :-) — 00010000, Dec 27 '09 at 19:26
Can you post your entire hook function code? The problem could hide not in the ToAscii itself, but in the surrounding hook logic. Also, as a wild guess, try `SetLastError(0)` at the end of the hook function. — atzz, Dec 28 '09 at 17:40
@atzz: I just added the hook function code. I believe it is ToAscii() that's the problem, because if I don't call it, the problem doesn't occur. Thanks for the SetLastError(0) suggestion, but unfortunately that didn't do anything. — 00010000, Dec 28 '09 at 18:41
@jdigital: Oops. Fixed. GetKeyboardState() was called in my real code, though. — 00010000, Dec 31 '09 at 03:59

score 4 · Answer 1 · answered Jun 19 '16 at 17:19

4

Quite an old thread. Unfortunately it didn't contain the answer I was looking for and none of the answers seemed to work properly. I finally solved the problem by checking the MSB of the MapVirtualKey function, before calling ToUnicode / ToAscii. Seems to be working like a charm:

if(!(MapVirtualKey(kbHookData->vkCode, MAPVK_VK_TO_CHAR)>>(sizeof(UINT)*8-1) & 1)) {
    ToAscii((UINT)kbHookData->vkCode, kbHookData->scanCode,
        keyboard_state, &wCharacter, 0);
}

Quoting MSDN on the return value of MapVirtualKey, if MAPVK_VK_TO_CHAR is used:

[...] Dead keys (diacritics) are indicated by setting the top bit of the return value. [...]

answered Jun 19 '16 at 17:19

Kristian Kraljic

826
12
11

could you please elaborate more? I don't get your code, what if you don't enter the if()? doing nothing? – Gianluca Ghettini Jan 28 '17 at 00:08
1

In case it's not entering the if, the current key pressed is a dead key. Depending on what you would like to achieve, you can react to it, or simply ignore it, in case you would like to only listen to non-dead key presses. Hope this helps. – Kristian Kraljic Jan 29 '17 at 16:18
Thank you, I got it – Gianluca Ghettini Jan 29 '17 at 16:20
1

Does not work for dead chars that use "ALT GR" key for example (right ALT, as it produces "CTRL + ALT" key message). Exemple of french keyboard layout trying to add accent this accent over a letter : ` = ì – Flox Jun 05 '17 at 18:42

DJm00n · Answer 2 · 2022-12-16T15:32:59.160

It is known that ToUnicode() and its older counterpart ToAscii() can change keyboard state of the current thread and thus mess with dead keys and ALT+NUMPAD keystrokes:

As ToUnicodeEx translates the virtual-key code, it also changes the state of the kernel-mode keyboard buffer. This state-change affects dead keys, ligatures, alt+numpad key entry, and so on. It might also cause undesired side-effects if used in conjunction with TranslateMessage (which also changes the state of the kernel-mode keyboard buffer).

To avoid that you can do your ToUnicode() call in a separate thread (it will have a separate keyboard state) or use a special flag in wFlags param that is documented in ToUnicode() docs:

If bit 2 is set, keyboard state is not changed (Windows 10, version 1607 and newer)

Or you can prepare sc->char mapping table beforehand and update it on language change event.

I think it should work with ToAscii() too but better not use this old ANSI codepage-dependant method. Use ToUnicode() API instead that can even return ligatures and UTF-16 surrogate pairs - if keyboard layout have them. Some do.

See Asynchronous input vs synchronous input, a quick introduction for the reason behind this.

This should be the accepted answer, even if it is only valid as of Windows 10. It's definitely the way to go for anyone looking for a solution now. — manu_unter, Dec 14 '22 at 16:40

score 3 · Answer 3 · answered Dec 28 '09 at 16:51

3

stop using ToAscii() and use ToUncode()
remember that ToUnicode may return you nothing on dead keys - this is why they are called dead keys.
Any key will have a scancode or a virtual key code but not necessary a character.

You shouldn't combine the buttons with characters - assuming that any key/button has a text representation (Unicode) is wrong.

So:

for input text use the characters reported by Windows
for checking button pressed (ex. games) use scancodes or virtual keys (probably virtual keys are better).
for keyboard shortcuts use virtual key codes.

answered Dec 28 '09 at 16:51

sorin

161,544
178
535
806

I tried using ToUnicode(), but the same thing happens... the "dead-key" is destroyed, and so a double-accent is shown when I press the accent key, and I can't make any accented letters. Uninstalling the hook immediately fixes the issue. Not calling ToAscii() or ToUnicode() immediately fixes the issue as well. – 00010000 Dec 28 '09 at 18:15
Try to make a copy of keyboard state structure in order to prevent Windows from altering the original one. – sorin Dec 31 '09 at 16:41

score 2 · Answer 4 · edited Jul 04 '12 at 08:08

2

Calling the ToAscii or ToUnicode twice is the answer. I found this and converted it for Delphi, and it works!

cnt:=ToUnicode(VirtualKey, KeyStroke, KeyState, chars, 2, 0);
cnt:=ToUnicode(VirtualKey, KeyStroke, KeyState, chars, 2, 0); //yes call it twice

edited Jul 04 '12 at 08:08

nhahtdh

55,989
15
126
162

answered Jul 04 '12 at 07:37

Martin

29
1

Its a little better as my '^' char is displayed only after the second key press, as expected, but doesn't work because the accent doesn't go over the letter as it should produce 'ô', but only produces 'o' – Flox Jun 04 '17 at 09:42
Actually your answer is the same as @Blue eyes below :https://stackoverflow.com/a/3625039/2165463 – Flox Jun 04 '17 at 09:43

score 2 · Answer 5 · answered Sep 02 '10 at 08:19

2

Call 'ToAscii' function twice for a correct processing of dead-key, like in:

int ta = ToAscii((UINT)kbHookData->vkCode, kbHookData->scanCode,
                 keyboard_state, &wCharacter, 0);
int ta = ToAscii((UINT)kbHookData->vkCode, kbHookData->scanCode,
                 keyboard_state, &wCharacter, 0);
If (ta == -1)
 ...

answered Sep 02 '10 at 08:19

Blue eyes

29
1

4

This just 'kills' the dead key for me. E.g. 'ë' becomes 'e', but pressing a dead key twice still gives e.g. ¨¨ – Steven Jeuris Aug 26 '13 at 11:40

Linoy Belchich · Answer 6 · 2019-10-27T08:52:09.460

1

I encountered this issue while creating a key logger in C# and none of the above answers worked for me.

After a deep blog searching, I stumbled across this keyboard listener which handles dead keys perfectly.

edited Oct 27 '19 at 08:52

answered Oct 27 '19 at 08:44

Linoy Belchich

58
9

score 1 · Answer 7 · answered Feb 21 '20 at 12:30

Here is a full code which covers dead keys and shortcut keys using ALT + NUMPAD, basically a full implementation of a TextField input handling:

    [DllImport("user32.dll")]
    public static extern int ToUnicode(uint virtualKeyCode, uint scanCode, byte[] keyboardState, [Out, MarshalAs(UnmanagedType.LPWStr, SizeConst = 64)] StringBuilder receivingBuffer, int bufferSize, uint flags);

    private StringBuilder _pressCharBuffer = new StringBuilder(256);
    private byte[] _pressCharKeyboardState = new byte[256];

    public bool PreFilterMessage(ref Message m)
    {
        var handled = false;

        if (m.Msg == 0x0100 || m.Msg == 0x0102)
        {

            bool isShiftPressed = (ModifierKeys & Keys.Shift) != 0;
            bool isControlPressed = (ModifierKeys & Keys.Control) != 0;
            bool isAltPressed = (ModifierKeys & Keys.Alt) != 0;
            bool isAltGrPressed = (ModifierKeys & Keys.RMenu) != 0;

            for (int i = 0; i < 256; i++)
                _pressCharKeyboardState[i] = 0;

            if (isShiftPressed)
                _pressCharKeyboardState[(int)Keys.ShiftKey] = 0xff;

            if (isAltGrPressed)
            {
                _pressCharKeyboardState[(int)Keys.ControlKey] = 0xff;
                _pressCharKeyboardState[(int)Keys.Menu] = 0xff;
            }

            if (Control.IsKeyLocked(Keys.CapsLock))
                _pressCharKeyboardState[(int)Keys.CapsLock] = 0xff;

            Char chr = (Char)0;

            int ret = ToUnicode((uint)m.WParam.ToInt32(), 0, _pressCharKeyboardState, _pressCharBuffer, 256, 0);

            if (ret == 0)
                chr = Char.ConvertFromUtf32(m.WParam.ToInt32())[0];
            if (ret == -1)
                ToUnicode((uint)m.WParam.ToInt32(), 0, _pressCharKeyboardState, _pressCharBuffer, 256, 0);
            else if (_pressCharBuffer.Length > 0)
                chr = _pressCharBuffer[0];

            if (m.Msg == 0x0102 && Char.IsWhiteSpace(chr))
                chr = (Char)0;


            if (ret >= 0 && chr > 0)
            {

            //DO YOUR STUFF using either "chr" as special key (UP, DOWN, etc..) 
            //either _pressCharBuffer.ToString()(can contain more than one character if dead key was pressed before)
            //and don't forget to set the "handled" to true, so nobody else can use the message afterwards

            }
        }

        return handled;
    }

score 0 · Answer 8 · answered May 27 '15 at 15:22

I copy the vkCode in a queue and do the conversion from another thread

@HOOKPROC
def keyHookKFunc(code,wParam,lParam):
    global gkeyQueue
    gkeyQueue.append((code,wParam,kbd.vkCode))
    return windll.user32.CallNextHookEx(0,code,wParam,lParam)

This has the advantage of not delaying key processing by the os

Amit Chauhan · Answer 9 · 2021-06-24T13:58:20.217

This works for me

byte[] keyState = new byte[256];

//Remove this if using
//GetKeyboardState(keyState); 

//Add only the Keys you want
keysDown[(int)Keys.ShiftKey] = 0x80; // SHIFT down
keysDown[(int)Keys.Menu] = 0x80; // ALT down
keysDown[(int)Keys.ControlKey] = 0x80; // CONTROL down
  
//ToAscii should work fine         
if (ToAscii(myKeyboardStruct.VirtualKeyCode, myKeyboardStruct.ScanCode, keyState, inBuffer, myKeyboardStruct.Flags) == 1)
{
    //do something
}

ToAscii/ToUnicode in a keyboard hook destroys dead keys

9 Answers9

Linked