How to stop further processing global hotkeys in C#

Question

I am trying to create a C# application that will react on a global hotkey; ALT+H and then when I release the ALT key. This actually does work very well but my problem is that when my application has done whatever it should do with the hotkey then it should stop this hotkey from being processed by other applications. I got most of my code below from this StackOverflow post, Global keyboard capture in C# application, but I have seen somewhere else that return (IntPtr)1; should be able to stop further processing of the key.

But... when I am in e.g. Word or Wordpad and I press ALT+H then Word shows all kind of menus - I don't want it to as I only want my application to do something :-)

I am sorry for this rather long code but I assume it is important so you can get the full overview. There is only 1 place where I use return (IntPtr)1;.

My code:

using System;
using System.Diagnostics;
using System.Runtime.InteropServices;
using System.Windows.Forms;
using System.Windows.Input;
    
namespace MyNewProgram
{
    static class Program
    {
    
        // Enable hotkeys
        // https://stackoverflow.com/a/604417/2028935
        private const int WH_KEYBOARD_LL = 13;
        private const int WM_KEYDOWN = 0x0100;
        private const int WM_KEYUP = 0x0101;
        private const int WM_SYSKEYUP = 0x0105;
        private const int VK_SHIFT = 0x10;
        private const int VK_MENU = 0x12;
        private static LowLevelKeyboardProc _proc = HookCallback;
        private static IntPtr _hookID = IntPtr.Zero;

        // Other variables
        public static bool isAltPressedInThisApp = false;

        private static MainWindow MyMainWindow;

        // --------------------------------------------------------------------------------------

        [STAThread] // STAThreadAttribute indicates that the COM threading model for the application is single-threaded apartment, https://stackoverflow.com/a/1361048/2028935

        static void Main()
        {
            // Hook application to keyboard
            _hookID = SetHook(_proc);

            Application.EnableVisualStyles();
            Application.SetCompatibleTextRenderingDefault(false);
            MyMainWindow = new MainWindow();
            Application.Run(MyMainWindow);

            // Unhook application from keyboard
            UnhookWindowsHookEx(_hookID);
        }

        // --------------------------------------------------------------------------------------
        // Required functions for globally hooking the keyboard

        [DllImport("user32.dll", CharSet = CharSet.Auto, SetLastError = true)]
        public static extern short GetKeyState(int keyCode);

        [DllImport("user32.dll", CharSet = CharSet.Auto, SetLastError = true)]
        private static extern IntPtr SetWindowsHookEx(int idHook, LowLevelKeyboardProc lpfn, IntPtr hMod, uint dwThreadId);

        [DllImport("user32.dll", CharSet = CharSet.Auto, SetLastError = true)]
        [return: MarshalAs(UnmanagedType.Bool)]
        private static extern bool UnhookWindowsHookEx(IntPtr hhk);

        [DllImport("user32.dll", CharSet = CharSet.Auto, SetLastError = true)]
        private static extern IntPtr CallNextHookEx(IntPtr hhk, int nCode, IntPtr wParam, IntPtr lParam);

        [DllImport("kernel32.dll", CharSet = CharSet.Auto, SetLastError = true)]
        private static extern IntPtr GetModuleHandle(string lpModuleName);

        private delegate IntPtr LowLevelKeyboardProc(int nCode, IntPtr wParam, IntPtr lParam);

        // --------------------------------------------------------------------------------------
        // Hook the keyboard - action

        private static IntPtr HookCallback(int nCode, IntPtr wParam, IntPtr lParam)
        {
            // React on KEYDOWN
            if (nCode >= 0 && ((wParam == (IntPtr)WM_KEYDOWN) || (wParam == (IntPtr)WM_SYSKEYUP)))
            {
                int vkCode = Marshal.ReadInt32(lParam);

                // H
                if ((Keys)vkCode == Keys.H)
                {
                    isAltPressedInThisApp = true;

                    // Is ALT pressed down
                    if ((GetKeyState(VK_MENU) & 0x8000) != 0)
                    {
                        Console.WriteLine("ALT + H");
                        return (IntPtr)1; // do not allow others to hook this key combo
                    }
                }
            }

            // React on KEYUP
            if (nCode >= 0 && wParam == (IntPtr)WM_KEYUP)
            {
                int vkCode = Marshal.ReadInt32(lParam);

                // Is ALT not pressed down
                if ((Keys)vkCode == Keys.LMenu)
                {
                    Console.WriteLine("ALT UP");
                }
            }

            return CallNextHookEx(_hookID, nCode, wParam, lParam);
        }

        // --------------------------------------------------------------------------------------
        // Hook the keyboard

        private static IntPtr SetHook(LowLevelKeyboardProc proc)
        {
            using (Process curProcess = Process.GetCurrentProcess())
            using (ProcessModule curModule = curProcess.MainModule)
            {
                return SetWindowsHookEx(WH_KEYBOARD_LL, proc, GetModuleHandle(curModule.ModuleName), 0);
            }
        }
    }
}

I am not a solid C# developer as I am taking my first few babysteps here - so why would I throw myself in the deep waters of global hotkeys as one of the first things ;-) Is there anyone that can give some hints here as I probably have a silly mistake somewhere?

Answer 1

The idea behind, whenever the alt key pressed, it starts to swallow keys into one local list. if the pattern we need to see has been seen, it won't send, but for the every other pattern, it starts to send the keys again in the order it receives.

the code for examination phase:

    enum WM
    {
        WM_KEYDOWN = 0x0100,
        WM_KEYUP = 0x0101,
        WM_SYSKEYUP = 0x0105,
        WM_SYSKEYDOWN = 0x0104,
    }

    private static IntPtr HookCallback(int nCode, IntPtr wParam, IntPtr lParam)
    {
        if (nCode < 0)
            return CallNextHookEx(_hookID, nCode, wParam, lParam);

        KBDLLHOOKSTRUCT kbd = (KBDLLHOOKSTRUCT)Marshal.PtrToStructure(lParam, typeof(KBDLLHOOKSTRUCT));
        Enum.TryParse<Keys>($"{kbd.vkCode}", out Keys key);
        Enum.TryParse<WM>($"{wParam}", out WM message);

        Console.WriteLine($"{message}:{key}");

        return CallNextHookEx(_hookID, nCode, wParam, lParam);
    }

when pressing alt+h and releasing, output:

WM_SYSKEYDOWN:LMenu
WM_SYSKEYDOWN:H
WM_SYSKEYUP:H
WM_KEYUP:LMenu

as you can see, windows sent both alt and h key. the code you provided only catch H key, so the window which receives keyboard messages think that alt pressed.

no one can fetch and filter out previously pressed key, so we need to catch alt key whenever we see it is pressed. if the next key not H, we should send the keys we have fetched in the order we received.

i have written the below code to handle the situation, but i can not be pretty sure how it works on the real windows machine cause i have an osx operating system just because of that i am running windows in the virtual machine which is also change key strokes when i'm pressing.

if it's still not enough, you can wait and i may be able to try and figure it out at my office in the real windows machine. but i think you get the idea and work it out on your own.

    [StructLayout(LayoutKind.Sequential)]
    public class KBDLLHOOKSTRUCT
    {
        public uint vkCode;
        public uint scanCode;
        public KBDLLHOOKSTRUCTFlags flags;
        public uint time;
        public UIntPtr dwExtraInfo;
    }

    [Flags]
    public enum KBDLLHOOKSTRUCTFlags : uint
    {
        LLKHF_EXTENDED = 0x01,
        LLKHF_INJECTED = 0x10,
        LLKHF_ALTDOWN = 0x20,
        LLKHF_UP = 0x80,
    }
    [DllImport("user32.dll")]
    static extern void keybd_event(byte bVk, byte bScan, uint dwFlags, UIntPtr dwExtraInfo);

    enum WM
    {
        WM_KEYDOWN = 0x0100,
        WM_KEYUP = 0x0101,
        WM_SYSKEYUP = 0x0105,
        WM_SYSKEYDOWN = 0x0104,
    }

    private static void ReSendKeys(int index = -1)
    {
        _index = -1;

        var copiedKeys = _swallowedKeys.ToArray();
        for (int i = 0; i < copiedKeys.Length; ++i)
        {
            bool up = copiedKeys[i].Item1 == (IntPtr)WM.WM_SYSKEYUP || copiedKeys[i].Item1 == (IntPtr)WM.WM_KEYUP;
            keybd_event((byte)copiedKeys[i].Item2.vkCode, (byte)copiedKeys[i].Item2.scanCode, up ? 2u : 0u, UIntPtr.Zero);
        }

        _index = index;
        _swallowedKeys.Clear();
    }

    private static List<Tuple<IntPtr, KBDLLHOOKSTRUCT>> _swallowedKeys = new List<Tuple<IntPtr, KBDLLHOOKSTRUCT>>();

    private static int _index = 0;
    private static IntPtr HookCallback(int nCode, IntPtr wParam, IntPtr lParam)
    {
        if (nCode < 0)
            return CallNextHookEx(_hookID, nCode, wParam, lParam);
        
        KBDLLHOOKSTRUCT kbd = (KBDLLHOOKSTRUCT)Marshal.PtrToStructure(lParam, typeof(KBDLLHOOKSTRUCT));
        Enum.TryParse<Keys>($"{kbd.vkCode}", out Keys key);
        Enum.TryParse<WM>($"{wParam}", out WM message);

        Console.Write($"{message}:{key}");

        // we know that when _index is -1, ReSendKeys function has been called
        // so do not filter out first alt key
        if (_index == -1)
        {
            _index++;
            Console.WriteLine();
            return CallNextHookEx(_hookID, nCode, wParam, lParam);
        }

        // we are at the beginning of the sequence we will catch
        // if it's alt key filter it out, and increment the variable
        if (_index == 0)
        {
            _swallowedKeys.Add(new Tuple<IntPtr, KBDLLHOOKSTRUCT>(wParam, kbd));

            if (message == WM.WM_SYSKEYDOWN && key == Keys.LMenu)
            {
                _index++;

                Console.WriteLine(" filtered out");
                return (IntPtr)1;
            }
            else
            {
                _swallowedKeys.Clear();
                // do nothing
            }
        }

        if (_index == 1)
        {
            _swallowedKeys.Add(new Tuple<IntPtr, KBDLLHOOKSTRUCT>(wParam, kbd));

            // if the next key is H, then filter it out also
            if (message == WM.WM_SYSKEYDOWN && key == Keys.H)
            {
                _index++;
                _swallowedKeys.RemoveAt(_swallowedKeys.Count - 1);

                Console.WriteLine(" filtered out");
                return (IntPtr)1;
            }
            // if not, we filtered out wrong sequence, we need to resend them
            else
            {
                Console.WriteLine();
                ReSendKeys();
                return (IntPtr)1;
            }
        }

        if (_index == 2)
        {
            _swallowedKeys.Add(new Tuple<IntPtr, KBDLLHOOKSTRUCT>(wParam, kbd));

            if (message == WM.WM_SYSKEYUP && key == Keys.H)
            {
                _index++;
                _swallowedKeys.RemoveAt(_swallowedKeys.Count - 1);

                Console.WriteLine(" filtered out");
                return (IntPtr)1;
            }
            else
            {
                // if user pressed H but not released and pressed another key at the same time
                // i will pass that situation, if u need to handle something like that, you got the idea, please fill that block of code
            }
        }

        if (_index == 3)
        {
            _swallowedKeys.Add(new Tuple<IntPtr, KBDLLHOOKSTRUCT>(wParam, kbd));

            if (message == WM.WM_KEYUP && key == Keys.LMenu)
            {
                _index = 0;
                _swallowedKeys.Clear();

                Console.WriteLine(" filtered out");
                Console.WriteLine("shortcut disabled");
                return (IntPtr)1;
            }
            else
            {
                Console.WriteLine();
                // user has been pressed Alt + H, H released but not alt, so we can expect one H again, but we need to send this pressed key with Alt prefixed
                ReSendKeys(1);
                return (IntPtr)1;
            }
        }

        Console.WriteLine();
        return CallNextHookEx(_hookID, nCode, wParam, lParam);
    }