0

I have been using a pre-compiled DLL in some software for a few years now. Now that software required a new feature so I dusted off my project and made my changes. It was originally built for .NET 4.0, but since that is unsupported I switched to .NET 4.8, trying to keep the same major version to avoid too many differences.

My original code I did not specify a calling convention so I assume StdCall was used, but when I went to run it I got a "PInvoke has left the stack unbalanced" exception. I changed all call types to Cdecl, which resolved that error. I think this was correct to do, but it could be related to my problem still.

Now one particular method is called, (one that is supposed to give a human readable reason for an error code) the application closes. If I try to execute the method in the watch window I get an error that the target process exited with a seemingly random code (-1073740940) which does not seem like the intention of the DLL.

Since the DLL hasn't changed, I assume something about dllimport has that is causing this issue. The documentation for the DLL denotes a single long input and a char* return type which I assume means it is an ANSI string, but I've tried multiple return marshal types. This is what I think is the most correct DLLImport I've tried (the DLL I'm using is not public so I've changed the DLL name and entry point).:

[DllImport("provided.dll", EntryPoint = "getErrorString", CallingConvention = CallingConvention.Cdecl)]
[return: MarshalAs(UnmanagedType.LPStr)]
public static extern string GetDLLErrorString(long errorCode);
Fr33dan
  • 4,227
  • 3
  • 35
  • 62
  • Does the process architecture (X86, X64) match that of the native DLL at runtime? – Oliver Weichhold Oct 18 '22 at 21:27
  • @OliverWeichhold Yes, both are x86. I do not have a x64 version of the DLL and thus made sure to build my application as 32 bit. – Fr33dan Oct 18 '22 at 21:37
  • The default calling convention did not change between .NET versions -- it was and still is `stdcall` on Windows. It would help if you had the *actual* definition of the function with its *actual* calling convention, as it now seems you're sort of throwing stuff at the wall to see what sticks. Do you have a header containing a prototype declaration of the function? Also, when you say you changed "all" call types, is that meant to imply you're calling more functions than just this one? A stack imbalance could also be the result of an incorrect declaration elsewhere. – Jeroen Mostert Oct 18 '22 at 21:40
  • @JeroenMostert Yes I am aware that the default convention didn't change, but how the [error of an incorrect convention is handled](https://stackoverflow.com/a/6768223) did. I wish I did have more detailed info about the DLL but all I've got is the DLL itself, and a small PDF that provides only the information included already in the question. – Fr33dan Oct 18 '22 at 21:44
  • It's quite possible the code was broken before, but that just didn't surface due to the runtime being more permissive or not enabling Managed Debugging Assistants by default. In the latter case you can make the problem "disappear" by disabling them again (specifically the PInvokeStackImbalance MDA) though that will just mean you have silent stack corruption as before. ...yeah, basically what the linked question is saying, which appeared after I typed my comment. :) – Jeroen Mostert Oct 18 '22 at 21:44
  • OK, so what do the docs for the DLL state *exactly*? What is the documented function prototype? I would expect something like `__declspec(dllexport) char* getErrorString(long errorCode)`, or if leaves out the `__declspec`, it should be saying something about calling conventions elsewhere. In addition, does it document anything about how strings are allocated and freed? – Jeroen Mostert Oct 18 '22 at 21:46
  • The documentation says this: `char* getErrorString(long errcode);` INPUTS: errcode: long. Contains the error code to look up. OUTPUTS: Return value points to string (char*) with error message. RETURN: char* to error string. Do not modify this string. Note that the return value is non-standard – Fr33dan Oct 18 '22 at 21:49
  • There is not other information in the documentation regarding calling conventions. Table of contents with the list of function names, then the functions outlined like shown above. – Fr33dan Oct 18 '22 at 21:50
  • Well, that's not good. One thing you could consider is using the `NetFx40_PInvokeStackResilience` element (also documented in the linked question) with your existing code, as-is, to let the runtime fix up the imbalances as it did before. Otherwise you're potentially looking at reviewing every single call you're making to this DLL to verify they're correct, not merely this one. – Jeroen Mostert Oct 18 '22 at 21:54
  • Disassemble or decompile the library and check what its export is really doing and what the calling convention is. – Ray Oct 18 '22 at 22:00
  • Code was always broken. But the new version of .net detected this and told you. – David Heffernan Oct 18 '22 at 22:03
  • @Oliver of course the architecture matches because otherwise we wouldn't be able to load the dll and call the function – David Heffernan Oct 18 '22 at 22:03
  • 2
    According to [Native interoperability best practices](https://learn.microsoft.com/en-us/dotnet/standard/native-interop/best-practices#common-windows-data-types) for a `LONG` in C, one should use `int` in C#. Also see [Windows Data Types](https://learn.microsoft.com/en-us/windows/win32/winprog/windows-data-types) and [Built-in types (C# reference)](https://learn.microsoft.com/en-us/dotnet/csharp/language-reference/builtin-types/built-in-types) – Tu deschizi eu inchid Oct 18 '22 at 22:03
  • Another problem is that C++ long on Windows is 32 but but C# long is 64 bit. And yet another is that the marshaler attempts to deallocate the return vakue by calling CoTaskMemFree. Was the string allocated by the COM allocator. – David Heffernan Oct 18 '22 at 22:04
  • The following may be helpful: [PInvoke for C function that returns char *](https://stackoverflow.com/a/61120111/10024425) – Tu deschizi eu inchid Oct 18 '22 at 22:16
  • I've tried adding `NetFx40_PInvokeStackResilience` to my runtime configuration, and changing the parameter type even going to far as to add `[MarshalAs(UnmanagedType.I4)]` to the parameter. I was able to find documentation outside the previously mentioned PDF stating that they do use `_declspec(dllexport)` for each function and that it was built with VS2008. Assuming they didn't change it, the default for VS2008 seems to be Cdecl so that is what I think I should be using. So far it still always exits when run. – Fr33dan Oct 18 '22 at 23:40
  • @DavidHeffernan I had to step away, come back and re-read everything with fresh eyes to fully process your comments and one of your ideas was it. It is the attempted deallocation by the marshaler that causes the closure. I don't know how the DLL is allocating the string, but if I return and `IntPtr` and use `Marshal.PtrToStringAnsi` it works. The `IntPtr` is consistant with input code per execution so I don't even think there is a memory leak issue by not deallocating. – Fr33dan Oct 19 '22 at 04:02
  • Right. But you should check whether or not you should be deallocating the returned string, and if so how. – David Heffernan Oct 19 '22 at 06:25
  • @DavidHeffernan without it being documented how it was allocated, how could I determine if/how to deallocate the string? – Fr33dan Oct 19 '22 at 13:22
  • Not easy. You could look for some example code in C++ or some other language that would show you the pattern. You could contact the developer. You could disassemble the code. – David Heffernan Oct 19 '22 at 14:36
  • @DavidHeffernan Good idea, I hadn't looked at the C++ sample code in years and forgot some was provided. It seems the statement "Do not modify this string" also means to not deallocate it as they do not in their sample code. – Fr33dan Oct 19 '22 at 22:38
  • Good. There's your answer. – David Heffernan Oct 20 '22 at 02:22

0 Answers0