SetWindowsHookEx not injecting DLL

Question

All I'm trying to do is get my DLL injected into some other programs (at process creation time) and get it to execute the DllMain function. Unfortunately, no matter what code I am trying, it never works.

For example, I have this code: http://pastebin.com/P4NzLb3X Basically it is just using SetWindowsHookEx to install a keyboard hook. Checking with Process Hacker however shows me that the DLL never actually gets injected into any process. :(

I searched already for the entire day for a solution. How can I solve this?

Your code writes status messages. Which message are you seeing? — Logicrat, Aug 23 '14 at 15:54
@Logicrat These: [link](http://i.imgur.com/mFuEvck.png) (i.e. everything successful) — Luxalpa, Aug 23 '14 at 16:03
Nice downvotes for no reason, but is actually someone able to answer this rather important question?! — Luxalpa, Aug 23 '14 at 18:14
I would suggest that you write some status messages from inside your DLL so that you will be able to tell how far execution has gotten inside the DLL/ Either that, or use a debugger to step thru the DLL's code. — Logicrat, Aug 23 '14 at 18:44
I did, but as I wrote above, the DLL doesn't get loaded, the attachment never gets executed (except during the LoadLibrary inside the program that injects the hook), and process monitoring software confirms that the DLL never gets applied to any process. — Luxalpa, Aug 23 '14 at 19:00
The injection doesn't occur until you put focus on a GUI program of the same bitness and then press a key. — Raymond Chen, Aug 23 '14 at 19:52
Thanks, I didn't know that! Unfortunately, whenever I press a button in such a program, the program freezes until I unload my hook. Both the Hook method as well as the DllMain function never get executed. http://pastebin.com/5kXmzu9C — Luxalpa, Aug 23 '14 at 20:40
If the program freezes, clearly *something* is happening. Use Process Monitor (available from the MS web site) to confirm whether the DLL is being loaded or not, or better still use a debugger to see exactly what happens. You can avoid breaking your debugger by hooking only a specific thread, or by putting the program being debugged (and the hooking program) onto a separate desktop; hooking a specific thread would be simpler. — Harry Johnston, Aug 24 '14 at 01:57
It's freezing because you're doing *far* too much work in the DllMain function. You cannot call all those functions. Do consider reading [the documentation](http://msdn.microsoft.com/en-us/library/windows/desktop/ms682583.aspx) — Cody Gray - on strike, Aug 24 '14 at 07:33
I used Process Monitor and can confirm that the DLL is NOT being loaded. Reducing the DllMain to only "return TRUE;" doesn't affect the result. — Luxalpa, Aug 24 '14 at 08:32

score 2 · Accepted Answer · edited May 23 '17 at 10:30

2

I solved the problem with the help of these two links:

http://www.gamedev.net/topic/568401-problem-with-loading-dll--setwindowshookex/

Global Keyhook on 64-Bit Windows

3 things had to be fixed:

Add a .def file for the DLL and use the exports there as Visual Studio obviously likes to make weird name mangling with extern "C" __declspec(dllexport) int - this fixes DLL loads on 32 bit processes
Add CALLBACK attribute to function (extern "C" int CALLBACK meconnect(...)) - this fixes crash that happens on 32 bit processes after the fix above.

Add a message loop into the host process (the process that is calling the SetWindowsHookEx function) like so:

MSG msg;
while(1) {
    // Keep pumping...
    PeekMessage(&msg, nullptr, 0, 0, PM_REMOVE);
    TranslateMessage(&msg);
    DispatchMessage(&msg);
    Sleep(10);
    SHORT v = GetKeyState(VK_RETURN);
    bool pressed = (v & 0x8000) != 0;
    if(pressed) 
        break;
}

this fixes the freeze in 64 bit processes.

edited May 23 '17 at 10:30

Community

1
1

answered Aug 24 '14 at 09:43

Luxalpa

430
4
15

1

Note that the first problem should have been caught by your code, since it checks for `NULL` returned by `GetProcAddress`. Why didn't you mention that? Note also that the first two problems were not in the code that you posted. – Raymond Chen Aug 24 '14 at 14:06
The first problem hasn't been caught by my code. I can not explain why as I don't know myself, I assume it is a bug somewhere in Visual Studio. As for the source code, I provided the source for the DLL in the comments to the OP as I didn't believe that it was related to the problem (since according to various process monitors the DLL never got even opened, I naively assumed that the DLL stuff never got executed). I simply missed that there could be an issue with exporting functions. – Luxalpa Aug 24 '14 at 19:18
2

I doubt you found a bug in Visual Studio – Raymond Chen Aug 24 '14 at 19:21
Fine, then it is just weird behaviour in Visual Studio. It is however clearly not working well. Apparently some module is okay with working with decorated functions, while a deeper or later module says "no." I can reproduce this problem just fine. If you don't put the exports into the .def and instead put a __declspec(dllexport) attribute to your function, the result will be a "decorated C function", which will be (im)properly recognized/resolved by GetProcAddress but apparently not by the actual injecting function (although that function shouldn't even know about that name...). – Luxalpa Aug 24 '14 at 21:01

SetWindowsHookEx not injecting DLL

1 Answers1