Exception when closing a Win32 dialog box in dll (from WPF application)

Question

I have a C# application that is calling a C++ function from a DLL. The C++ function shows a dialog box and a exit button only.

The C++ function in DLL is looking like:

//the exported function for clients to call in DLL
HINSTANCE hInstance;
__declspec(dllexport) int __cdecl StartDialog(string inString)
{
    hInstance = LoadLibrary(TEXT("My.dll"));
    DialogBox(hInstance, MAKEINTRESOURCE(ID_DLL_Dialog), NULL, DialogProc);
    return 1;
}

BOOL CALLBACK DialogProc(HWND hwnd, UINT message, WPARAM wParam, LPARAM lParam)
{
    switch (message)
    {
    case WM_COMMAND:
        switch (LOWORD(wParam))
        {
        case IDD_BUTTON_EXIT:
            DestroyWindow(hwnd);
            return TRUE;            
        }
    }
    return FALSE;
}

If I call my StartDialog function in a simple C++ program, it works. I can show the dialog, and can close it properly when I click exit in my dialog.

typedef int(__cdecl *StartDialogFunc)(string);
StartDialogFunc StartDialog = nullptr;
HINSTANCE hDll = LoadLibrary(TEXT("My.dll"));
StartDialog = (StartDialogFunc)GetProcAddress(hDll, "StartDialog");

int i = StartDialog("hello");    //this is working
cout << i;

If I call it in my C# application, after I click the exit button, the dialog closes, and give me an exception. The C# code looks like:

[DllImport("My.dll", CallingConvention = CallingConvention.Cdecl)]
public static extern int StartDialog(string s);

int i = StartDialog("hello");    //this causes a exception

The error message is like:

Debug Assertion Failed!

Program: (some paths...) My.dll

File: d:\program files (x86)\microsoft visual studio 14.0\vc\include\xmemory0

Line: 100

Expression: "(_Ptr_user & (_BIG_ALLOCATION_ALIGNMENT - 1)) == 0" && 0

For information on how your program can cause an assertion failure, see the Visual C++ documentation on asserts.

How can I know what is exactly wrong in my DLL?

Try changing the C++ function signature to take a `WCHAR*`, as C++'s `string` is incompatible with C#. Also, to get a handle to the current DLL, use GetModuleHandle(NULL), not LoadLibrary. — user1610015, Nov 30 '16 at 08:28
Ding ding ding, user1610015 is exactly correct on all counts. The CLR cannot marshal a `String` object as a C++ `std::string` object. @user1610015, please consider upgrading your comment to an answer. — Cody Gray - on strike, Nov 30 '16 at 09:37

score 1 · Accepted Answer · answered Nov 30 '16 at 09:42

1

Try changing the C++ function signature to take a WCHAR*, as C++'s std::string is incompatible with C#. Also, to get a handle to the current DLL, use GetModuleHandle(NULL), not LoadLibrary.

HINSTANCE hInstance;

__declspec(dllexport) int __cdecl StartDialog(const WCHAR* inString)
{
    hInstance = GetModuleHandle(NULL);
    DialogBox(hInstance, MAKEINTRESOURCE(ID_DLL_Dialog), NULL, DialogProc);
    return 1;
}

answered Nov 30 '16 at 09:42

user1610015

6,561
2
15
18

Great! A char array is working for me. But `hInstance = GetModuleHandle(NULL);` is not working, my old method works though. – Dia Nov 30 '16 at 09:58
True, I didn't know this until now but GetModuleHandle(NULL) returns the handle of the EXE and not the DLL. As this ( http://stackoverflow.com/questions/119706/how-to-get-my-own-codes-module-handle answer says, the correct solution is to write a DllMain function and store the hInstance parameter into a global variable. – user1610015 Nov 30 '16 at 10:15
Thanks. Now I successfully get HINSTANCE from my Dll main function. – Dia Dec 01 '16 at 03:15

Exception when closing a Win32 dialog box in dll (from WPF application)

1 Answers1