mingw DLL from C#: why do I have to override new/delete?

Question

I'm trying to call a minimal C function from C# on Windows 10. I use mingw/g++ to compile the C code into a .dll

It turns out that I have to define opterator new[] or compile the .dll using Visual Studio. Otherwise my C# program crashes with the following error:

The program '[14740] Test.exe' has exited with code -1073741819 (0xc0000005) 'Access violation'.

I'd really love to understand what exactly is happening here and how I can resolve this issue without overriding all the new/delete operators but still using mingw.

Here's the minimal example reproducing the error including a workaround (if AddNewOperator is defined operator new[] will be defined and the resulting .dll will work fine):

Test.cs (compiled/run with Visual Studio 2017):

using System;
using System.Runtime.InteropServices;
class Program
{
    [DllImport("libTest", CallingConvention = CallingConvention.StdCall, ExactSpelling = true)]
    public static extern int TestFunction();
    static void Main(string[] args)
    {
        Console.WriteLine("!!" + TestFunction());
    }
}

Test.cpp compiled with mingw (see below):

#include <new>
#include <cstdlib>

#ifdef AddNewOperator // This will fix the issue
void* operator new[](std::size_t sz){
    return std::malloc(sz);
}
#end

extern "C" {
int __stdcall __declspec(dllexport) TestFunction() {
        int* test = new int[3]; // removing this line will make everything work when building
        return test[2];
}

And here's the build script:

# Remove the following # and the compiled dll will work just fine
g++ -g -s -Wall -c -fmessage-length=0 Test.cpp  #-DAddNewOperator
g++ -g -shared -o libTest.dll *.o -Wl,--subsystem,windows

Edit: compiling everything for x86 instead of 64 bit also fixes the issue (which is again no option for me)

Answer 1

TL;DR

You must not mix allocation / deallocation between compilers!

The problem you are facing is quite tricky and actually your program should crash every time, with or without the void* operator new[](size_t){...} definition.

If you debug your program, it actually should crash while deleting your test variable. This variable is created using mingw's new operator, but deleted using MSVC delete operator, and they are not interoperable. So you have to use mingw's delete function.

for a simple test you can just do:

c++ code:

int* test = nullptr;
int __stdcall __declspec(dllexport) TestFunction() {
    test = new int[3]; // note test is global
    return test[2];
}
void __stdcall _declspec(dllexport) CleanUp() {
    delete[] test;
}

c# code:

public static extern int TestFunction();
public static extern int CleanUp();
static void Main(string[] args)
{
    Console.WriteLine("!!" + TestFunction());
    CleanUp();
}

---

Why is your program not crashing if you redefine the new operator?!

I actually don't know for sure, but i think, the malloc implementation of mingw uses legacy C runtime which uses HeapAlloc for allocating and HeapFree for deleting your test variable. In short, you are simply lucky/unlucky it is not crashing when you custom defined your operator new and used malloc inside...

However, if you compile it using Visual Studio, both (dll and exe) are using the same runtime, so allocation/deallocation is done within the same memory space organizer. BUT still it is UB and you will run into problems! E.g.: If you create your libary with msvc10 and want to use this library with msvc14 same can happen here! I can remember some issues with code coming from a bug where the memory also was managed wrong; We used a library created with msvc11 but our code was compiled with msvc12...