D3D->CreateDevice() fails inside an injected DLL

Question

I'm writing a DLL that hooks certain D3D-functions by changing needed pointers in the VMT of the injected process. The algorythm is like:

1. Obtain a dummy window handle.
2. Init a dummmy D3D-object, create a D3D-device.
3. Grab the needed offsets (VMT - D3D base address).
4. Destroy the D3D object and the window handle.
5. Get the D3D*.dll base of the injected process.
6. Construct the actual VMT pointers (offsets + base).
7. Search for them in the process heap.
8. If found - change the pointers to my own.

It works totally fine within the application, where I init D3D, start the drawing loop and then fire the hooking procedure, but it doesn't work from a DLL - for some reason it crashes while trying to create a dummy device with last error code = 126 ("The specified module could not be found", though D3D*.dll loads correctly).

I know that at the same moment there can be only one D3DDevice inited, but in my test app I can create more of 1 - only 1 works, but I still can grab the VMT.

So, the question is - what is the difference between running the same code from EXE and from DLL?

Answer 1

I had the same problem. What I thought it could be is that we need to create a device using the same thread that created the first one (main thread).

I tried all different combinations of parameters while creating the device. I tried different DX9 applications/games.

I found that the main thread hypothesis isn't correct. I updated the code of a DX sample and added my own code to run on a secondary thread and it worked without any problems. I have got a few more ideas that I will try. and update back

The problem was happening to me because I was calling CreateDevice() from DllMain, according to Dll Best Practices it is forbidden to call any function that may have threads crossing over each other, this will cause a deadlock which is what happens here. The solution is to follow this answer here.