simple windows callback thunks for x64

Question

Many of you are familiar with ATL thunks e.g. for window creation. The class that makes this work CStdCallThunk is targetting the WindowProc call. In essense it turns the global callback into a member function of a C++ object.

This type of thunk won't work for a SetWindowsHookEx callback which needs the first parameter intact. For 32 bit windows I found a tidy solution in CAuxThunk, part of the ATL/AUX library. Unfortunately this is not working for a native 64 bit executable

I wonder if there is any x64 assembly guru that can patch this CAuxThunk to work for 64 bit windows or come up with any equivalent thunk that would turn a this __stdcall callback into a member function?

LRESULT CALLBACK CallWndProc(int nCode, WPARAM wParam, LPARAM lParam)

Thanks,
nikos

This sounds quite wrong-headed, you should only ever use a hook for windows that are not yours. — Hans Passant, Oct 28 '13 at 12:02

Yakov Galka · Answer 1 · 2021-06-23T17:23:51.333

I described a generic way to generate any thunk code you want in this answer. Let's redo it for your case, as an exercise.

Suppose that your class is defined as:

struct YourClass {
    LRESULT YourMemberFunc(int nCode, WPARAM wParam, LPARAM lParam);
};

Write your thunk in C++ with placeholders for the actual addresses:

LRESULT CALLBACK CallWndProc(int nCode, WPARAM wParam, LPARAM lParam) {
    YourClass *x = reinterpret_cast<YourClass*>(0x1122112211221122);
    __int64 im = 0x3344334433443344;
    LRESULT (YourClass::*m)(int,WPARAM,LPARAM) = *reinterpret_cast<LRESULT (YourClass::**)(int,WPARAM,LPARAM)>(&im);
    return (x->*m)(nCode, wParam, lParam);
}

And call it in a way that prevents the compiler from inlining the call:

int main() {
    LRESULT (CALLBACK *volatile fp)(int, WPARAM, LPARAM) = CallWndProc;
    fp(0, 0, 0);
}

Compile in release and see the generated assembly (in Visual Studio, see the assembly window during debugging and turn on "Show code bytes"):

4D 8B C8                       mov         r9,r8  
4C 8B C2                       mov         r8,rdx  
8B D1                          mov         edx,ecx  
48 B9 22 11 22 11 22 11 22 11  mov         rcx,1122112211221122h  
48 B8 44 33 44 33 44 33 44 33  mov         rax,3344334433443344h  
48 FF E0                       jmp         rax

This is going to be your thunk, with 44 33 44 33 44 33 44 33 replaced with the pointer to your member (&YourClass::YourMemberFunc) and 22 11 22 11 22 11 22 11 replaced with the pointer to the actual object instance, at run-time.

Explanation of what's going on in the thunk

In the x64 calling convention (of which there is only one on Windows), the first four parameters are passed in the rcx, rdx, r8, r9 registers, in this order, from left to right. So when our thunk gets called, we have

rcx = nCode, rdx = wParam, r8 = lParam

For member functions there is an implicit first parameter holding this pointer, so when entering YourMemberFunc we must have

rcx = this, rdx = nCode, r8 = wParam, r9 = lParam

The compiler generated code does exactly this adjustment: it shifts r8 -> r9, rdx -> r8, ecx -> edx, and then assigns our placeholder this = 1122112211221122 to rcx. Now it has the parameters set up, and it can continue with an indirect jump to the function itself. rax is used to hold the return value, so it does not have to be preserved across function calls. This is why it is used here to temporarily hold the destination address, which gives an opportunity for tail call optimization (a call/return pair replaced with single jump).

Why we have to do an indirect call? Because otherwise we will get a relative jump. But we cannot use a hard-coded relative jump because the thunk is going to be copied to different addresses in memory! Therefore, we resort to setting the absolute address at runtime and doing an indirect jump instead.

HTH

score 0 · Answer 2 · answered Oct 28 '13 at 10:22

0

Because in 64 bit mode the low-level conventions are the same for all types (stdcall, cdecl and thiscall) you don't really need assembly code to achieve this. Just create a static function that calls the appropriate member function. You will need to figure out the proper this pointer to use, for example through associating the hwnd in the lparam with an object. If you only have a single callback this would be no problem of course. Something like:

LRESULT CALLBACK static CallWndProc(int nCode, WPARAM wParam, LPARAM lParam)
{
    Window w = GetWindowByHwnd(((CWPSTRUCT*)lParam)->hwnd);
    return w->CallWndProc(nCode, wParam, lParam);
}

answered Oct 28 '13 at 10:22

Jester

56,577
4
81
125

such a hypothetical GetWindowByHwnd would require some sort of singleton or other means of looking up window handles with C++ classes. I am trying to avoid all that, hence the need for an elegant low level thunk – nikos Oct 28 '13 at 12:48
I don't see how the low-level thunk would figure out which instance to call if you have multiple ones. – Jester Oct 28 '13 at 12:53
1

You could use `GetWindowLongPtr` to get the this pointer. I remember having seen a thunk solution for x64 though, I will try to find that one again. – Alexander Gessler Oct 28 '13 at 13:49

score 0 · Answer 3 · edited May 23 '17 at 12:01

0

Updated See Below

Take a look at SetWindowLongPtr http://msdn.microsoft.com/en-us/library/windows/desktop/ms644898(v=vs.85).aspx

This allows you to associate a pointer with the Window. Given that WindowProc looks like this

LRESULT CALLBACK WindowProc( _In_ HWND hwnd, _In_ UINT uMsg, _In_ WPARAM wParam, _In_ LPARAM lParam );

You can use the hwnd parameter with GetWindowLongPtr to retrieve the this pointer in your callback.

Updated

Take a look at How to thunk a function in x86 and x64? (Like std::bind in C++, but dynamic)

This seems to be what you are looking for

edited May 23 '17 at 12:01

Community

1
1

answered Oct 28 '13 at 13:50

John Bandela

2,416
12
19

as I say in my opening post, the issue is not with window procedures but with hooking. Window procedures are nicely handled by ATL thunking. What I need is something for a non-window class. I don't have a window to associate with it, it is irrelevant – nikos Oct 29 '13 at 06:07
@nikos how many thunks are you installing? Is it a bounded number, or can be anything? – John Bandela Oct 29 '13 at 11:58
thanks for your updated post and link, that looks promising but I have to test it first (I haven't got a clue what the code is doing, let's see if it works :). I don't use many of these hooks, typically while I'm doing something with a window I may instantiate a temporary hook (wrapped in a tidy C++ object), finish the work and then uninstall the hook (destroy the object) – nikos Oct 29 '13 at 16:50
@nikos are the hooks overlapping in lifetime. Do you do 1 hook, finish with it and then another, or are there potentially multiple hooks running at the same time? – John Bandela Oct 29 '13 at 17:34
they are overlapping some times – nikos Oct 30 '13 at 06:10
ok, I think I made it work, please check the [linked post](http://stackoverflow.com/questions/12136309/how-to-thunk-a-function-in-x86-and-x64-like-stdbind-in-c-but-dynamic). If you understand x64 assembly please check I didn't do anything silly :) – nikos Nov 10 '13 at 15:19

simple windows callback thunks for x64

3 Answers3

Explanation of what's going on in the thunk