How to debug DLLMain function

Question

I want to write DLL but I want to test what I am writing.
I tried to debug it with F5 but I receive an error:

I read the article Walkthrough: Creating and Using a Dynamic Link Library (C++) how to do it and its latest version.
But they suggest to create header file that contains the functions.

In my case, I created DLL project (Loader) with dllmain.cpp.

// dllmain.cpp : Defines the entry point for the DLL application.
#include "stdafx.h"

BOOL APIENTRY DllMain( HMODULE hModule,
                       DWORD  ul_reason_for_call,
                       LPVOID lpReserved
                     )
{
    HANDLE hd;
    switch (ul_reason_for_call)
    {
    case DLL_PROCESS_ATTACH:
        hd = CreateFileA("C:\\Users\\myuser\\Desktop\\test.dll", GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL);
        OutputDebugString(L"HELLO");
    case DLL_THREAD_ATTACH:
    case DLL_THREAD_DETACH:
    case DLL_PROCESS_DETACH:
        break;
    }
    return TRUE;
}

I created a new C++ console project in the same solution but I don't know how to call the function DllMain and debug it.

Just to make it organized, I have now two projects:
1. Loader - this is the DLL project
2. DLLTester - this is a console application that will run the DLL from the Loader project.

I added the folder of the DLL project (Loader) to the Additional Include Directories of the DLLTester project.
I created a header (Loader.h) to the DLL project and added the function signature DLLMain.

Now I can see it in the DLLMain function.
But I don't have idea what arguments I need to pass to this function:

I understand that I need to pass 3 arguments:
1. HMODULE hModule
2. DWORD ul_reason_for_call
3. LPVOID lpReserverd

But I don't what I need to enter there.

Code:
Loader project:
Loader.h:

#pragma once
#include "stdafx.h"

BOOL APIENTRY DllMain(HMODULE hModule,
    DWORD  ul_reason_for_call,
    LPVOID lpReserved
);

dllmain.cpp:

// dllmain.cpp : Defines the entry point for the DLL application.
#include "stdafx.h"

BOOL APIENTRY DllMain( HMODULE hModule,
                       DWORD  ul_reason_for_call,
                       LPVOID lpReserved
                     )
{
    HANDLE hd;
    switch (ul_reason_for_call)
    {
    case DLL_PROCESS_ATTACH:
        hd = CreateFileA("C:\\Users\\myuser\\Desktop\\test.dll", GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL);
        OutputDebugString(L"HELLO");
    case DLL_THREAD_ATTACH:
    case DLL_THREAD_DETACH:
    case DLL_PROCESS_DETACH:
        break;
    }
    return TRUE;
}

DLLTester project:
DLLTester.cpp:

// DLLTester.cpp : Defines the entry point for the console application.
//

#include "stdafx.h"
#include "Loader.h"

int main()
{
    DllMain(?,?,?) -> not sure what to enter here
    return 0;
}

How it looks:

It is not a sensible question, DllMain is called by the operating system. Happens as soon as the OS loader finishes loading and relocating the DLL in memory. Calling it yourself is very unlikely to come to a good end. And is not necessary for testing, if the test program does not have an implicit dependency on the DLL (it probably should) then simply use LoadLibrary(). — Hans Passant, Dec 25 '17 at 12:55
@HansPassant OK, can you suggest me how to debug such DLL ? This DLL has the `DLLMain` function and I need to check that it enters the `DLL_PROCESS_ATTACH` case. I want to do it with step over function of Visual Studio. — E235, Dec 25 '17 at 13:01
Just set a breakpoint on the code you want to debug, the VS debugger does not have a problem with a break in DllMain. Why you'd favor StepOver is not obvious, that will never step into DllMain. Why you are talking about debugging and the question is about testing is not obvious either btw. — Hans Passant, Dec 25 '17 at 13:16
But this is the problem. When I run F5 I receive "Unable to start program \Loader.dll. \Loader.dll is not a valid Win32 application". When I am checking it on regular C++ project, no error appear. It is only on DLL projects. — E235, Dec 25 '17 at 13:19
You have to start an EXE program, it can load a DLL. Project > Properties > Debugging > Command setting, pick a EXE file that exercise the DLL. Or simply add another project to the solution, a console mode app that uses the DLL, select it as the startup project. — Hans Passant, Dec 25 '17 at 13:23
Notice that in your Solution Explorer the DLL Project ("Loader") is in **bold** typeface, meaning it is selected as the StartUp project. You can't run a DLL, hence the error message. You need to right-cick on "DLLTester" and choose "*Select as StartUp Project*". — rustyx, Dec 25 '17 at 21:10

score 3 · Answer 1 · answered Dec 25 '17 at 13:48

DllMain is actually a fairly special function that is automatically called when a DLL is loaded using LoadLibrary or LoadLibraryEx (unless you do some evil stuff to prevent it).

If you really want to call it, you can use GetProcAddress, but believe it or not, a DLL's actual entry point will normally be _DllMainCRTStartup (which actually calls DllMain in addition to doing things like calling constructors and doing other initialization). It can even be overridden by the developer to be something completely different.

So I don't really see how calling DllMain can do anything useful for any normal DLL. It doesn't perform all of the initialization/cleanup (notably the C++ constructor calls at startup), and it's already being called (unless you suppress it with DONT_RESOLVE_DLL_REFERENCES).

score 0 · Answer 2 · answered Dec 26 '17 at 06:27

The easy way to debug DllMain is to use static linking (that is with __declspec(dllexport)/__declspec(dllimport) syntax), and to then set your DLL project as the startup project but have the debugger start the .exe that links to the library. You set the program to run in the project settings (Project->Properties then Debugging -> Command.

You can then set breakpoints and do traces as normal from your DLL.

I know you can do breakpoints and whatnot for normal functions in a dynamic-loaded DLL I'm just not sure about DllMain.

How to debug DLLMain function

2 Answers2