ExecutionEngineException on LowLevel Keyboard Hook

Question

I'm currently trying to get Global LowLevel keyboard hooks working in .net 5 with wpf, but after some time I always get an ExecutionEngineException and my App crashes. According to the docs, this exception should never occur at all anymore, but for me it does. It is also not specific to my project, as a minimal WPF .net 5 project will also crash after some key mashing.

Has anyone any idea how to either fix it or get around this problem?

Example Code for the Main Window:

public partial class MainWindow : Window {
    private const int WH_KEYBOARD_LL = 13;
    private const int WM_KEYDOWN = 0x0100;
    private const int WM_SYSKEYDOWN = 0x0104;

    private delegate IntPtr LowLevelKeyboardProc(int nCode, IntPtr wParam, IntPtr lParam);

    [DllImport("user32.dll", CharSet = CharSet.Auto, SetLastError = true)]
    private static extern IntPtr SetWindowsHookEx(int idHook, LowLevelKeyboardProc lpfn, IntPtr hMod,
        uint dwThreadId);

    [DllImport("user32.dll", CharSet = CharSet.Auto, SetLastError = true)]
    private static extern IntPtr CallNextHookEx(IntPtr hhk, int nCode,
        IntPtr wParam, IntPtr lParam);
    
    [DllImport("kernel32", SetLastError = true, CharSet = CharSet.Ansi)]
    static extern IntPtr LoadLibrary([MarshalAs(UnmanagedType.LPStr)] string lpFileName);


    private IntPtr _hook;


    public bool Hook() {
        if (_hook == IntPtr.Zero) {
            var modPtr = LoadLibrary("user32.dll");

            _hook = SetWindowsHookEx(WH_KEYBOARD_LL, Handler, modPtr, 0);
        }

        return _hook != IntPtr.Zero;
    }

    private IntPtr Handler(int code, IntPtr param, IntPtr lParam) {
        return CallNextHookEx(_hook, code, param, lParam);
    }

    public MainWindow() {
        InitializeComponent();
        Hook();
    }
}

The Callstack I get:

WindowsBase.dll!System.Windows.Threading.Dispatcher.GetMessage(ref System.Windows.Interop.MSG msg, System.IntPtr hwnd, int minMessage, int maxMessage)
WindowsBase.dll!System.Windows.Threading.Dispatcher.PushFrameImpl(System.Windows.Threading.DispatcherFrame frame)
WindowsBase.dll!System.Windows.Threading.Dispatcher.PushFrame(System.Windows.Threading.DispatcherFrame frame)
WindowsBase.dll!System.Windows.Threading.Dispatcher.Run()
PresentationFramework.dll!System.Windows.Application.RunDispatcher(object ignore)
PresentationFramework.dll!System.Windows.Application.RunInternal(System.Windows.Window window)
PresentationFramework.dll!System.Windows.Application.Run()
EngineExecutionError.dll!EngineExecutionError.App.Main()

Does .NET's default calling convention match the calling convention spelled out by the contract of the low-level keyboard hook API? — IInspectable, Sep 08 '21 at 12:23
Also, what's the rationale behind passing the module handle of *user32.dll* as the 3rd argument to `SetWindowsHookEx`? — IInspectable, Sep 08 '21 at 12:30
Rationale for user32.dll: https://stackoverflow.com/questions/17897646/setwindowshookex-fails-with-error-126/17898148#17898148 I had GetModuleHandle on the current module before with the same results, it "works" too so I didn't change it back — CShark, Sep 08 '21 at 12:36
Both are wrong. Read the [documentation](https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-setwindowshookexw). — IInspectable, Sep 08 '21 at 12:38
The default Calling convention is WinApi which refers to stdcall on windows which is the default for the win32 api (https://learn.microsoft.com/en-us/cpp/cpp/stdcall?view=msvc-160) — CShark, Sep 08 '21 at 12:38
You can't have a .NET callback function that's not fixed in some way as .NET will move it when it wants. You should either use a GCHandle or make Handler static and build a reference on it (something like `static LowLevelKeyboardProc _handler = Handler` and use _handler, not Handler). — Simon Mourier, Sep 08 '21 at 12:48
Well. if I set it to IntPtr.Zero the same thing happens. GetModuleHandle is called on the module that implements the handler; anyway, the KEYBOARD_LL should not need a module handle to the handler dll, because the handler is always called in the thread that registered it (https://learn.microsoft.com/en-us/previous-versions/windows/desktop/legacy/ms644985(v=vs.85)#remarks) — CShark, Sep 08 '21 at 12:48
@SimonMourier looks like the GC was at fault here, totally forgot that... If you make an answer, I'll accept it — CShark, Sep 08 '21 at 13:09
You should answer to correspond to your question exactly :-) — Simon Mourier, Sep 08 '21 at 13:33

score 4 · Accepted Answer · answered Sep 08 '21 at 14:38

As pointed out by the comments, you should not pass a lambda or delegate directly into a Win32 Api call (or any C api call for that matter). The fix is either to make a static variable containing a reference to the function

private static LowLevelKeyboardProc _handler = Handler;
[...]
_hook = SetWindowsHookEx(WH_KEYBOARD_LL, _handler, modPtr, 0);

or to pin the delegate, so the CLR won't move it around, which is what I opted for.

private LowLevelKeyboardProc _handler;
private GCHandle _gcHandler;

public KeyboardHook() {
    _handler = Handler;
    _gcHandler = GCHandle.Alloc(_handler);
}

[...]

_hook = SetWindowsHookEx(WH_KEYBOARD_LL, _handler, modPtr, 0);

For the low level hooks, you seem to be able to set modPtr to either LoadLibrary("user32.dll"), GetModuleHandle(null) or IntPtr.Zero. Seems to work with all of them.

ExecutionEngineException on LowLevel Keyboard Hook

1 Answers1