11

I've seen posts talk about what might cause differences between Debug and Release builds, but I don't think anybody has addressed from a development standpoint what is the most efficient way to solve the problem.

The first thing I do when a bug appears in the Release build but not in Debug is I run my program through valgrind in hopes of a better analysis. If that reveals nothing, -- and this has happened to me before -- then I try various inputs in hopes of getting the bug to surface also in the Debug build. If that fails, then I would try to track changes to find the most recent version for which the two builds diverge in behavior. And finally I guess I would resort to print statements.

Are there any best software engineering practices for efficiently debugging when the Debug and Release builds differ? Also, what tools are there that operate at a more fundamental level than valgrind to help debug these cases?

EDIT: I notice a lot of responses suggesting some general good practices such as unit testing and regression testing, which I agree are great for finding any bug. However, is there something specifically tailored to this Release vs. Debug problem? For example, is there such a thing as a static analysis tool that says "Hey, this macro or this code or this programming practice is dangerous because it has the potential to cause differences between your Debug/Release builds?"

Eric
  • 706
  • 5
  • 13
  • 1
    With VC, you can instrument release builds with debug information and debug these, too. There's no real fun in stepping through optimized code, but you do have a pretty good chance to find bugs that way. – sbi Jun 22 '10 at 07:11
  • Right, I forget that you can compile debug information into the release version with gcc as well. – Eric Jun 23 '10 at 07:50

6 Answers6

4

One other "Best Practice", or rather a combination of two: Have Automated Unit Tests, and Divide and Conquer.

If you have a modular application, and each module has good unit tests, then you may be able to quickly isolate the errant piece.

djna
  • 54,992
  • 14
  • 74
  • 117
  • +1 Unit tests he lp indeed in two ways: (1) You don't have to do so much (release mode) debugging. (2) Debugging the unit test code often is a short cut to the source of the error. – ur. Jun 22 '10 at 07:28
3

The very existence of two configurations is a problem from debugging point of view. Proper engineering would be such that the system on the ground and in the air behave the same way, and achieve this by reducing the number of ways by which the system can tell the difference.

Debug and Release builds differ in 3 aspects:

  • _DEBUG define
  • optimizations
  • different version of the standard library

The best way around, the way I often work, is this:

  • Disable optimizations where performance is not critical. Debugging is more important. Most important is disable function auto-inlining, keep standard stack frame and variable reuse optimizations. These annoy debug the most.
  • Monitor code for dependence on DEBUG define. Never use debug-only asserts, or any other tools sensitive to DEBUG define.
  • By default, compile and work /release.
Pavel Radzivilovsky
  • 18,794
  • 5
  • 57
  • 67
  • 1
    I think this hits at the heart of the problem. When you think about it, techniques such as unit testing and regression testing try to localize bugs by bisecting the time and space that has elapsed between a working and a broken version. – Eric Jun 23 '10 at 01:18
  • But debugging the difference between a release and debug build should really occur by bisecting the differences between a debug and release build because you don't need unit tests if you just want to get the bug to appear in your debugger. As you mentioned, these include optimizations and differences in dependence. – Eric Jun 23 '10 at 01:19
3

When I come across a bug that only happens in release, the first thing I always look for is use of an uninitialized stack variable in the code that I am working on. On Windows, the debug C runtime will automatically initialise stack variables to a know bit pattern, 0xcdcdcdcd or something. In release, stack variables will contain the value that was last stored at that memory location, which is going to be an unexpected value.

Secondly, I will try to identify what is different between debug and release builds. I look at the compiler optimization settings that the compiler is passed in Debug and Release configurations. You can see this is the last property page of the compiler settings in Visual Studio. I will start with the release config, and change the command line arguments passed to the compiler one item at a time until they match the command line that is used for compiling in debug. After each change I run the program and reproducing the bug. This will often lead me to the particular setting that causes the bug to happen.

A third technique can be to take a function that is misbehaving and disable optimizations around it using the pre-processor. This will allow you run the program in release with the particular function compiled in debug. The behaviour of the program which has been built in this way will help you learn more about the bug.

#pragma optimize( "", off )
void foo() {
    return 1;
}
#pragma optimize( "", on )

From experience, the problems are usually stack initialization, memory scrubbing in the memory allocator, or strange #define directives causing the code to be compiled incorrectly.

Marc O'Morain
  • 3,699
  • 5
  • 28
  • 34
  • 2
    Another common issue I run in to is code that has a side affect in an assert() macro. The macro is compiled away in release, and the side affect never occurs. – Marc O'Morain Jun 22 '10 at 07:23
1

The most obvious cause is simply the use of #ifdef and #ifndef directives associated DEBUG or similar symbol that change between the two builds.

Before going down the debugging road (which is not my personal idea of fun), I would inspect both command lines and check which flags are passed in one mode and not the other, then grep my code for this flags and check their uses.

One particular issue that comes to mind are macros:

#ifdef _DEBUG_
  #define CHECK(CheckSymbol) { if (!(CheckSymbol)) throw CheckException(); }
#else
  #define CHECK(CheckSymbol)
#endif

also known as a soft-assert.

Well, if you call it with a function that has side effect, or rely on it to guard a function (contract enforcement) and somehow catches the exception it throws in debug and ignore it... you will see differences in release :)

Matthieu M.
  • 287,565
  • 48
  • 449
  • 722
0

When debug and release differ it means:

  1. you code depends on the _DEBUG or similar macros (defined when compiling a debug version - no optimizations)
  2. your compiler has an optimization bug (I seen this few times)

You can easily deal with (1) (code modification) but with (2) you will have to isolate the compiler bug. After isolating the bug you do a little "code rewriting" to get the compiler generate correct binary code (I did this a few times - the most difficult part is to isolate the bug).

I can say that when enabling debug information for release version the debugging process works ... (though because of optimizations you might see some "strange" jumps when running).

You will need to have some "black-box" tests for your application - valgrind is a solution in this case. These solutions help you find differences between release and debug (which is very important).

INS
  • 10,594
  • 7
  • 58
  • 89
  • 1
    From what I've seen most of the bugs that surface with optimizations turn on are not bugs in the optimizer, but in the code being compiled. The optimizations just make them show up. Things like missing `volatile` and `restrict` are likely to cause this type of problem. Modern respectable compilers do have bugs, but also go through a LOT of automated testing. – nategoose Jun 22 '10 at 15:32
0

The best solution is to set up something like automated unit testing to thoroughly test all aspects of the application (not just individual components, but real world tests which use the application the same way a regular user would with all of the dependencies). This allows you to know immediately when a release-only bug has been introduced which should give you a good idea of where the problem is.

Good practice to actively monitor and seek out problems beats any tool to help you fix them long after they happen.

However, when you have one of those cases where it's too late: too many builds have gone by, can't reproduce consistently, etc. then I don't know of any one tool for the job. Sometimes fiddling with your release settings can give a bit of insight as to why the bug is occurring: if you can eliminate optimizations which suddenly make the bug go away, that could give you some useful information about it.

Release-only bugs can fall into various categories, but the most common ones (aside from something like a misuse of assertions) is:

1) Uninitialized memory. I use this term over uninitialized variables as a variable may be initialized but still be pointing to memory which hasn't been initialized properly. For this, memory diagnostic tools like Valgrind can help.

2) Timing (ex: race conditions). These can be a nightmare to debug, but there are some multithreading profilers and diagnostic tools which can help. I can't suggest any off the bat, but there's Coverity Integrity Manager as one example.

stinky472
  • 6,737
  • 28
  • 27