4

I'm developing a C# application that calls into an unmanaged (C++) dll. I've found a certain user operation that consistently leads to the application crashing when run from Windows Explorer. However, when launched from the Visual Studio debugger, the crash does not occur. I therefore cannot step into the code at the time of the crash and debug exactly what's going on.

What might cause the binary to crash when run from Explorer, but not from Visual Studio? Note that I'm using a Release build; Debug builds crash neither in Visual Studio nor in Explorer.

(If it's relevant, I can say that the crash has to do with manipulating a malloc-allocated array in the C++ DLL. I tracked it down by painstakingly commenting out blocks of code, rebuilding, running from Windows, & checking whether or not the crash occurs. However, I've reached a point where it's become very difficult to proceed without being able to properly break in the debugger).

I'm just interested in being able to recreate the crash within Visual Studio.

Sinatr
  • 20,892
  • 15
  • 90
  • 319
J23
  • 3,061
  • 6
  • 41
  • 52
  • show your code, and maybe screenshot of error – David Abaev Jan 07 '15 at 08:57
  • Why are you using `malloc` in C++? Remember it only allocates the memory, it doesn't call the constructor of the object. You could change to `new`, and write some assertions to test your invariants. – Hame Jan 07 '15 at 08:58
  • Is your application multi-threaded? If so switching from debug to release can sometimes reveal race-conditions. Alternatively, try to write a c++ equivalent version of your C# code to see if there problem is in the inter-op part of the code. – Henrik Jan 07 '15 at 08:59
  • @David: Here's what the crash produces: http://screencast.com/t/ecDZ4K5pAXbq. The dll code comes from Seismic Unix (http://tinyurl.com/ntoyxcd), which I'm (partially) porting to Windows for use in my project. It's not easy to show the code as the original is a big spaghetti mess (i.e. the actual call to 'malloc' is buried many functions deep, different location from where it's used, etc). I'm more interested in getting it to crash in Visual Studio to debug the crash myself, than help with figuring out the crash. – J23 Jan 07 '15 at 09:08
  • @Hame: DLL is not my code, and the question I'm interested in is why the crash would occur when the same binary is launched from Explorer but not from Visual Studio. – J23 Jan 07 '15 at 09:09
  • @Henrik: I'm always using Release; Release crashes when launched from Explorer, but Release doesn't crash when launched from VS. As per previous comments, for the sake of this question I'm just interested in being able to recreate the crash within Visual Studio. – J23 Jan 07 '15 at 09:12
  • 1
    Use Tools + Attach to Process to attach the debugger after the process got started. You can also use Debugger.Launch() in your C# code. – Hans Passant Jan 07 '15 at 09:24
  • @HansPassant: Good idea! Unfortunately, while this *does* allow me to set breakpoints in the managed code, breakpoints in the dll are never hit. I'm certain unmanaged debugging is working in general (& Just My Code is disabled), as I can hit the breakpoints if I begin the debug session from within Visual Studio - they only aren't hit if I first launch the app and then attach. Thus, I'm still unable to debug this particular crash (as the crash *only* occurs if I start launch the app from outside VS). – J23 Jan 07 '15 at 09:49
  • Click the Select button in the attach dialog to select the debugging engines you want to use. – Hans Passant Jan 07 '15 at 09:50
  • You can also create a minidump:http://www.codeproject.com/Articles/1934/Post-Mortem-Debugging-Your-Application-with-Minidu – stamhaney Jan 07 '15 at 10:46
  • Publish the `DllImport`-s for C++ functions and lines with corresponding calls. – i486 Jan 07 '15 at 10:48
  • @HansPassant: Perfect! That did it. I assumed it was including native debugging by default, as it was checked when I first clicked that button - but I guess not. While a bit more cumbersome than working all from within the IDE, Debugger.Launch() has enabled mem to at last start digging into this issue. Thanks again! :) – J23 Jan 07 '15 at 23:59
  • @stamhaney: Great approach that I'd completely forgotten about. While it won't be necessary for *this* issue (Debugger.Launch() worked), that'll undoubtedly come in handy farther down the line, when my employer reports a crash that I can't reproduce. I'll keep it in mind :) – J23 Jan 08 '15 at 00:02
  • @i486: Please see my earlier comments. – J23 Jan 08 '15 at 00:02
  • VS runs `XXX.vshost.exe` which runs the real EXE file. Windows Explorer runs directly the EXE file. – i486 Jan 08 '15 at 00:05
  • Only if you have the Visual Studio hosting process enabled (I believe), which I don't. In any case, the issue was solved by launching the exe and using Debugger.Launch to attach to VS. The cause probably had to do with the different heap behaviors as documented here: http://blogs.msdn.com/b/reiley/archive/2011/08/28/side-effects-of-debugger.aspx – J23 Jan 08 '15 at 00:17
  • @HansPassant: If you have a moment, I've got a follow-up question regarding a peculiarity with using Debugger.Launch(), posted here: http://stackoverflow.com/questions/27831318/console-output-not-shown-in-visual-studio-when-using-debugger-launch. Specifically: debug sessions initiated that way do not show any console output in the Output pane - again making it difficult to debug as I normally would... – J23 Jan 08 '15 at 01:05
  • http://stackoverflow.com/a/2070532/17034 – Hans Passant Jan 08 '15 at 09:19
  • Hmm...except that whether or not I've got the Visual Studio hosting process enabled, it does show logs in the Output pane (as I'd like) when launched from within VS, and doesn't when using Debugger.Launch(). So this seems to not be it... – J23 Jan 08 '15 at 20:12

1 Answers1

7

When a program works in debug but crash in release, the problem is really often buffer overflow, so you should look for something like an incorrect buffer length variable.

About why it is not crashing when debugging, here is a small article about the side effects of debugger. As you can read, the heap can have a different behavior when a program is launched by a debugger. Buffer overflows often happen on the heap, and you said it might happen with a malloc-ed buffer, so that would be the reason.

Now, to make your program crash while debugging, the only way is probably to attach after launch. If setting breakpoints in the DLL project doesn't work, try to start the debugging with the DLL project instead, and specify your executable as DLL host. If you can't hit breakpoints like that, well, you can always set breakpoints in assembly code, that should always work, but isn't really practical. You can add a compiled assembly breakpoint using __debugbreak() on MSVC.

Note that differences might still happen even when attaching, a debugger will always slightly alter the target behavior. Though only undefined behaviors should change.

Edit: I missed that, but the article says that you can disable debug heap by setting an environment variable _NO_DEBUG_HEAP = 1. That might be the simplest solution to debug your problem (if it works).

ElderBug
  • 5,926
  • 16
  • 25
  • Excellent & thorough answer...thanks! Using a combination of this & comments on my original question, I was able to attach & reproduce this (and several other) runtime issues I was experiencing. Accepted. – J23 Jan 07 '15 at 23:56
  • For those who are interested, here's some more info on the debug heap: http://preshing.com/20110717/the-windows-heap-is-slow-when-launched-from-the-debugger/ – J23 Jan 08 '15 at 06:58