Strange situation while redirecting debug statements to stderr

Question

I am stuck in a situation. I am working on an embedded device (based on linux OS and ARM processor, 32 bit). It is a touch screen device that has many peripherals like smartcard reader, GPS, GPRS . I am coding in C and my application is crashing after some time when I am redirecting my debug statements as shown below. I have around 300 debug print statements that I am printing using this function like macro. The device can also be connected to the system terminal through a USB cable. When I am printing these debug statements in my system terminal, the application is not crashing anywhere, but when I am not using my computer's terminal and run the application in the device only then it crashes after some time:

#ifdef DEBUG_TEST
  #define DEBUG_TEST 1
  #else
  #define DEBUG_TEST 0
  #endif

  #define DEBUG_PRINT(fmt, ...) \
             do { if (DEBUG_TEST) fprintf(stderr, fmt, ##__VA_ARGS__); } while (0)

But when I am turning off these debug statements, the application doesn't crash anywhere. I don't understand why is it happening.
According to my guess, since the device does not have its own standard terminal, so printing these debug messages is creating a buffer due to which it crashes after some time, while on the other hand when I turn off the debug statements then it works fine. Please suggest why it might be happening?

How does it crash exactly? Segfault? Killed by kernel? Other? — fge, Jan 02 '12 at 19:15
Do you have the option to run this under GDB, or to collect a core dump? — Oliver Charlesworth, Jan 02 '12 at 19:20
If I had to guess, I'd say that the debug statements are not the problem, they're merely provoking a [heisenbug](http://en.wikipedia.org/wiki/Heisenbug). — Oliver Charlesworth, Jan 02 '12 at 19:21
It crashes only when i turn ON the debug statements and run it on device only without using my systems terminal(means without connecting device to my system ) ..therefore iam not able to trace where and why is it crashing ... — john, Jan 02 '12 at 19:23
Such bugs are often provoked by memory corruption. It's seems that debug output is not the issue. — Nikolay Polivanov, Jan 02 '12 at 19:23
@john: Then this it's going to be virtually impossible for anyone here to give a useful answer. I would strongly suggest collecting a core dump, which you can then examine *post hoc* in GDB to identify what is causing the crash. — Oliver Charlesworth, Jan 02 '12 at 19:25
Ok can u hint me where these debug statements will go because the device does not has a standard terminal that can be treated as a way for stdout/ stderr — john, Jan 02 '12 at 19:25
See this: http://stackoverflow.com/questions/2919378/how-to-enable-core-dump-in-my-linux-c-program. — Oliver Charlesworth, Jan 02 '12 at 19:27
@john: as p.kolya said, it would suggest a memory corruption, try running it under valgrind — Hubert Kario, Jan 02 '12 at 19:30
My personal guess is that some buffer internal to the stderr implementation is overflowing. I'd to freopen() stderr to write to a file, at least, when it isn't connected to some device. This should either avoid the problem or give a better indication where things are crashing. — Dietmar Kühl, Jan 02 '12 at 19:42
"It crashes only when i turn ON the debug statements and run it on device only without using my systems terminal" <-- maybe the problem is there: what is stderr defined to in this case? — fge, Jan 02 '12 at 19:42
yes you are pointing to the same thing in which i also have a doubt .. — john, Jan 02 '12 at 19:46

score 3 · Accepted Answer · answered Jan 02 '12 at 19:27

Perhaps one of the arguments to one of your DEBUG_PRINT calls is causing fprintf's string-interpolator routine to reference invalid memory? For example, something like this in your code would crash your app inside fprintf():

const char * badPointer = (const char *) 0xDEADBEEF;  // deliberately pointing to invalid memory
DEBUG_PRINT("Crashing now!  %s\n", badPointer);

... either that, or it's possible there is a bug in your system's stdio or USB implementation that is causing the crash.

To narrow things down, you could either try commenting out various DEBUG_PRINT statements until the crash goes away (at which point you can reasonably suspect that it was one of the now-commented-out DEBUG_PRINT statements that was causing the crash, and comment some back in until you figure out who the culprit is)... or if you suspect a bug in fprintf(), you could make a torture-test like this:

while(1) DEBUG_PRINT("I think I %s!\n", "can");

... and see if running that gives you the expected output (infinite output of the string) or crashes. If it crashes, then that would suggests a bug outside of your program.

score 1 · Answer 2 · answered Jan 02 '12 at 19:27

Is the only thing that you are changing these statements, or are you going from debug to release builds? Switching from a debug to a release build will change things like optimization, and memory padding, so it could be that your crash is happening in non debug because of these.

If all you are changing is the debug trace on and off, I would suggest you very carefully check all your format arguments (%s, %d, etc) and make sure you aren't passing a variable as the wrong type.

A recent GCC should, with `-Wall` or `-Wformat` alone, find the format arguments mismatch. — Basile Starynkevitch, Jan 02 '12 at 20:45

score 0 · Answer 3 · answered Jan 03 '12 at 13:35

How do you run your application without a controlling terminal ? You can try the following :

run your application with a terminal connected, but stderr redirected

./yourapp 2>/dev/null

or

./yourapp 2>somefile.log

and see if it still crashes. Don't forget to do ulimit -c unlimited, if you want a core file to be dumped.

If you have a busybox based userspace, you can try to use syslogd in circular buffer mode, and replace your fprintf(stderr, ...) by a call to syslog, and see if it still crashes.

Strange situation while redirecting debug statements to stderr

3 Answers3