Dll import Marshal C++ string issue

Question

Here is the signature of the DLL function I am trying to import

typedef char*(*DLLFUNC_Encrypt)(const char*, unsigned int, unsigned int&);

Here is my C# code

[DllImport("AuthCrypto.dll", EntryPoint = "Encrypt")]
public static extern IntPtr Encrypt(IntPtr data, int size, ref int mode);

public static string EncryptData(string data)
{
    int mode = 5;
    IntPtr dataIntPtr = Marshal.StringToHGlobalAnsi(data);
    IntPtr data_enc = CryptoAPI.Encrypt(dataIntPtr, data.Length, ref mode);

    return Marshal.PtrToStringAnsi(data_enc);
}

Here is my exception:

A call to PInvoke function 'CryptoAPI::Encrypt' has unbalanced the stack. This is likely because the managed PInvoke signature does not match the unmanaged target signature. Check that the calling convention and parameters of the PInvoke signature match the target unmanaged signature.

The native call uses a reference modifier which means this is C++ and not plain old C. PInvoke is based around plain old C. — JaredPar, Dec 11 '13 at 21:55
I would start by just writing a plain old C function that wraps the C++ code. The C function acts as a bridge between managed which prefers to see C code and the actual code which is written in C++ — JaredPar, Dec 11 '13 at 21:59
The only problem with that approach is this .dll is closed source. Is there another way? — patriot, Dec 11 '13 at 22:04
You could attempt to replace the last parameter with a pointer. — Collin Dauphinee, Dec 11 '13 at 22:06
It doesn't matter if you write `unsigned int&` or `unsigned int*`, that's only syntactic sugar. Both are pointers internally. And in C# both are represented by `ref int`. — Max Truxa, Dec 11 '13 at 22:09
Side note: I do not know what you want to encrypt, but note that your code does incorrect encryption for any input strings which contain characters not convertible to ASCII/ANSI. Since i don't know how and where the data is decrypted again, i don't want want to make suggestions. However, if non-ANSI characters are of concern, you might think about using Marshal.StringToHGlobalUni instead (and also decrypting the encrypted data into a Unicode string again). — , Dec 11 '13 at 22:56
Side note 2: Is Encrypt returning a pointer to a byte buffer, or a pointer to an actual string (character sequence -- which i speculate it is, since how would you determine the size/length of the returned byte buffer)? If it is a pointer to a byte buffer, is it ensured that it will not contain zero bytes except as a terminating zero-byte? Otherwise PtrToStringAnsi will not convert all bytes but stop when hitting the first zero-byte. — , Dec 11 '13 at 23:03
possible duplicate of [Why are Cdecl calls often mismatched in the "standard" P/Invoke Convention?](http://stackoverflow.com/questions/15660722/why-are-cdecl-calls-often-mismatched-in-the-standard-p-invoke-convention) — Hans Passant, Dec 11 '13 at 23:04
Side note 3 (last one): When writing/storing the string with the encrypted data using .NET functionality, pay attentention to the System.Text.Encoding in use. Most likely, you want Encoding.UTF-8 (which should be used for 8-bit character values; Encoding.ASCII is not recommended as it is only safe for 7-bit values.) — , Dec 11 '13 at 23:16

Max Truxa · Answer 1 · 2013-12-11T22:12:28.220

Looks like you need to specify the calling convention to use. Since on DLLFUNC_Encrypt none is set explicitly, the compiler uses its default. Which calling convention is considered being the default one depends on your compiler settings (for Visual Studio it is cdecl if you did not change it).

You can fix that by explicitly specifying the calling convention DLLFUNC_Encrypt like this:

typedef char*(*__cdecl DLLFUNC_Encrypt)(const char*, unsigned int, unsigned int&);

and adjusting your platform invoke accordingly:

[DllImport("AuthCrypto.dll", EntryPoint = "Encrypt", CallingConvention = CallingConvention.Cdecl)]
public static extern IntPtr Encrypt(IntPtr data, int size, ref int mode);

The calling conventions you can pass to DllImport are defined in the CallingConvention enumeration.

I just read that the DLL you are invoking is closed source. If you can't specify the calling convention in the DLL, you can either look if you find the calling convention used by the library in its documentation or you can just test until you find a working one (it's probably either StdCall or Cdecl).

Dll import Marshal C++ string issue

1 Answers1