How to debug such memory corruption?

Question

int main () {
       allocating_resource();
       call_other_libs();
       ...
       release_resource();
       return 0;
}

After the program runs, the main returns.

And after it accessed returne 0, the stack pointer points to a bad address in main(), then the executable crashed.

Program received signal SIGSEGV, Segmentation fault.

GI __libc_free(mem=0x3f21a843) at malloc.c:2020

I guess there are some illegal memory accesses, but the code base is too large to check. Review and analyse all the code is not realistic.

Disable some code is also unacceptable due to large code base.

With core dump there 's no hint I can use due to it's crashed at the main stack and after the return clause executed.

I know how to use gdb, but the project is so large that it seems to hard to find the root cause.

valgrind --tool=memcheck seems to no help.

How to solve such problem?

You are passing an invalid pointer to free, I suppose. Compile with debugging symbols and use a debugger (e.g. gdb on UNIX) and break on calls to free. — Jeremy, Dec 26 '15 at 19:57
There is no possible magic; if the code is big, searching for a bug is a difficult task. Track first which freeing cause the crash and then try to track back from that... — Jean-Baptiste Yunès, Dec 26 '15 at 20:07
see my updated answer. there is one more tool called valgrind that can detect memory related issues. This applies if your issue is memory related. — phoenix, Dec 26 '15 at 20:10
what about MemProf. I have added in answer. I know this may help you because it gives function by function and also detects allocated and unreferenced memory. — phoenix, Dec 26 '15 at 20:27
In case you are facing stack issues Valgrind's (experimental) stack checker tool (`exp-sgcheck`) might help you : http://valgrind.org/docs/manual/sg-manual.html — alk, Dec 26 '15 at 22:07
if valgrind is not able to pinppoint some memory allocation/free problem, then the most likely case is some allocated memory as be overrun, like an buffer input overrun or a array of pointers overrun. — user3629249, Dec 27 '15 at 16:15
Suggest a good first step would be replace the `call_other_libs(); ...` with a stub that does nothing, then see if the problem still exists. If the problem disappears, then the source of the problem is the call_other_libs() function. Similar such testing could be used to progressively add/remove code until the problem disappears, then narrow the debugging efforts to that small area that was skipped. To avoid a lot of editing, use the #if 0 ... #endif to cause a block of code to be skipped (after a re-compile/re-link/rerun — user3629249, Dec 27 '15 at 16:25
When compiling, always enable all the warnings, then fix all those warnings, (for gcc, at a minimum use: `-Wall -Wextra -pedantic` Suggest also using `-Wconversion` and `-std=c99`) — user3629249, Dec 27 '15 at 16:32

score 2 · Answer 1 · edited Jun 20 '20 at 09:12

2

You can use GDB. This stackoverflow link has details about how to debug using GDB. If you google, you can get many such helpful links on GDB.
You can also use valgrind, if you are sure about memory related issues.

There is one more memory profiler called MemProf. It gives memory allocated for each function and can also detect issues. See the link for details.
There are also c++ specific tools for memory profiling like:
mempro and MTuner. You can use trial version for free.

edited Jun 20 '20 at 09:12

Community

1
1

answered Dec 26 '15 at 19:59

phoenix

3,069
3
22
29

the log doesn't help, because the program crashed after the main return. – Dec 26 '15 at 20:20

score 0 · Answer 2 · edited May 23 '17 at 11:52

Since we are not having some kind of code access here, I'll have to assume that in some magic way free is called when the scope of main is destroyed (maybe the use of smart pointers? maybe some sophisticated macro definitions... can't really tell). I would try to recreate the problem in the following manner:

int main () 
{
       {
           allocating_resource();
           call_other_libs();
           ...
           release_resource();
       }
       return 0;
}

or

  int main()
  {
    mainhelper();
    return 0;
  }

where mainhelper will contain the main code.

Hopefully after those steps the problem will persist and the logs won't be completely damaged, as you suggest, because the program is terminating.

also, try to play with the optimization flags (more like in disabling it) and add -ggdb3 debug flag (assuming gcc here). Maybe it will help you in some bizarre way.

Some other posts concerning this matter, if you haven´t checked them out yet:

segmentation fault after main returns

Program receives SIGSEGV error after return 0

they are all stating more of the same: valgrind should be able to deliver an answer.

How to debug such memory corruption?

2 Answers2