0

I updated my project from VS2005 (targeting .net 2) to VS2010 (targeting .net4). It seems the pInvokeStackImbalance MDA is enabled by default, then I get a bunch of "unbalanced the stack" exceptions. Take this one for example: 

[DllImportAttribute("gdi32.dll")]
private static extern IntPtr CreateSolidBrush(BrushStyles enBrushStyle, int crColor);

It was working in .net 2 but now it throws exception. I changed it to this and it works: 

[DllImportAttribute("gdi32.dll", CallingConvention = CallingConvention.ThisCall)]
private static extern IntPtr CreateSolidBrush(BrushStyles enBrushStyle, int crColor);

To my surprise, pinvoke.net lists it as 

[DllImport("gdi32.dll")]
static extern IntPtr CreateSolidBrush(uint crColor);

Why my old is not working? It seems the pinvoke.net is wrong but how do I find out which calling conversion is it given a win32 function?

EDIT
My project is using the code from C# Rubber Rectangle to do the XOR drawing. Apparently the code needs a fix to work in .Net 4.

Community
  • 1
  • 1
AZ.
  • 7,333
  • 6
  • 44
  • 62

1 Answers1

3

CreateSolidBrush uses stdcall. Almost all Win32 APIs do so. They never use thiscall. The function is declared as:

HBRUSH CreateSolidBrush(
  __in  COLORREF crColor
);

so your mistake is simply that your version has too many parameters. The pinvoke.net declaration that you have found is correct.

The stdcall convention pushes params right to left which explains how your code worked even with an extra spurious parameter.

David Heffernan
  • 601,492
  • 42
  • 1,072
  • 1,490
  • How do you know it's `stdcall`? The link doesn't say anything about it. – AZ. Oct 08 '12 at 21:59
  • 2
    API functions always are. If you look it up in the header file you'll see it declared as `WINAPI` which evaluates to `__stdcall`. A very small number are `__cdecl` because they have variable number of arguments. For example [`wsprintf`](http://msdn.microsoft.com/en-us/library/windows/desktop/ms647550.aspx). The API docs will always explicitly call out a `__cdecl` API function. – David Heffernan Oct 08 '12 at 22:05
  • Thanks! Cannot believe the method signature was wrong all the time!! – AZ. Oct 08 '12 at 22:09
  • Obviously the `enBrushStyle` parameter you were passing was being ignored. One can only wonder at where your version came from! ;-) – David Heffernan Oct 08 '12 at 22:13
  • Yeah, they got it wrong. Of course, it should be obvious seeing that the function is called Create**Solid**Brush. The style is part of the name. Anyway, it's all clear now. – David Heffernan Oct 08 '12 at 22:17
  • The OP forgot to mention he ran his code on the 64-bit version of Windows. The default platform target got changed in VS2010. – Hans Passant Oct 08 '12 at 23:21
  • Yes it's in 64bit win7. What do you mean the default platform target got changed? – AZ. Oct 09 '12 at 22:28