5

I have a single threaded program that crashes consistently at certain points right after free() is called when running in non-debug mode.

When in debug mode however, debugger breaks on the line that calls free() even though there are no break points set. When I try to step to the next line again, debugger breaks again on the same line. Stepping once again resumes execution as normal. No crash, no segfault, nothing.

EDIT-1: Contrary to what I wrote above, crashes in non-debug mode turns out to be inconsistent, which makes me think I am somehow writing somewhere that I shouldn't. (Breaks in debug mode are still consistent, though.)

Call stack at the breaks shows some windows library functions(I think) called after the function that calls free() statement. I have no idea how to interpret them. And consequently, I have no idea how to go about debugging in this situation.

I have provided the call stacks at break points below. Can someone point me in a direction where I can tackle the problem? What might be causing the breaks in debugger mode?

Program is run on Windows Vista, compiled with gcc 4.9.2, debugger used is gdb. Assume double release is not the case.(I use ::operator new and ::operator delete overloads that catch that. Situation described is the same without these overloads as well.)

Note that the crash(or the involuntary breaks in debugger) is consistent. Happens every time, in the same execution point.

Here is the call stack at the initial break:

(Note that free_wrapper() is the function that houses free() statement that causes the crash/breaks.)

#0 0x770186ff   ntdll!DbgBreakPoint() (C:\Windows\system32\ntdll.dll:??)
#1 0x77082edb   ntdll!RtlpNtMakeTemporaryKey() (C:\Windows\system32\ntdll.dll:??)
#2 0x7706b953   ntdll!RtlImageRvaToVa() (C:\Windows\system32\ntdll.dll:??)
#3 0x77052c4f   ntdll!RtlQueryRegistryValues() (C:\Windows\system32\ntdll.dll:??)
#4 0x77083f3b   ntdll!RtlpNtMakeTemporaryKey() (C:\Windows\system32\ntdll.dll:??)
#5 0x7704bcfd   ntdll!EtwSendNotification() (C:\Windows\system32\ntdll.dll:??)
#6 0x770374d5   ntdll!RtlEnumerateGenericTableWithoutSplaying() (C:\Windows\system32\ntdll.dll:??)
#7 0x75829dc6   KERNEL32!HeapFree() (C:\Windows\system32\kernel32.dll:??)
#8 0x75a99c03   msvcrt!free() (C:\Windows\system32\msvcrt.dll:??)
#9 0x350000 ?? () (??:??)
--> #10 0x534020    free_wrapper(pv=0x352af0) (C:\dm\bin\codes\CodeBlocks\ProjTemp\src\Unrelated\MemMgmt.cpp:282)
#11 0x407f74    operator delete(pv=0x352af0) (C:\dm\bin\codes\CodeBlocks\ProjTemp\main.cpp:1002)
#12 0x629a74    __gnu_cxx::new_allocator<char>::deallocate(this=0x22f718, __p=0x352af0 "\nÿÿÿÿÿÿº\r%") (C:/Program Files/CodeBlocks/MinGW/lib/gcc/mingw32/4.9.2/include/c++/ext/new_allocator.h:110)
#13 0x6c2257    std::allocator_traits<std::allocator<char> >::deallocate(__a=..., __p=0x352af0 "\nÿÿÿÿÿÿº\r%", __n=50) (C:/Program Files/CodeBlocks/MinGW/lib/gcc/mingw32/4.9.2/include/c++/bits/alloc_traits.h:383)
#14 0x611940    basic_CDataUnit<std::allocator<char> >::~basic_CDataUnit(this=0x22f714, __vtt_parm=0x781df4 <VTT for basic_CDataUnit_TDB<std::allocator<char> >+4>, __in_chrg=<optimized out>) (include/DataUnit/CDataUnit.h:112)
#15 0x61dfa1    basic_CDataUnit_TDB<std::allocator<char> >::~basic_CDataUnit_TDB(this=0x22f714, __in_chrg=<optimized out>, __vtt_parm=<optimized out>) (include/DataUnit/CDataUnit_TDB.h:125)
#16 0x503898    CTblSegHandle::UpdateChainedRowData(this=0x353cf8, new_row_data=..., old_row_fetch_res=..., vColTypes=..., block_hnd=...) (C:\dm\bin\codes\CodeBlocks\ProjTemp\src\SegHandles\CTblSegHandle.cpp:912)
#17 0x502fcc    CTblSegHandle::UpdateRowData(this=0x353cf8, new_row_data=..., old_row_fetch_res=..., vColTypes=..., block_hnd=...) (C:\dm\bin\codes\CodeBlocks\ProjTemp\src\SegHandles\CTblSegHandle.cpp:764)
#18 0x443272    UpdateRow(row_addr=..., new_data_unit=..., vColTypes=..., block_hnd=..., seg_hnd=...) (C:\dm\bin\codes\CodeBlocks\ProjTemp\src\DbUtilities.cpp:910)
#19 0x443470    UpdateRow(row_addr=..., vColValues=..., vColTypes=...) (C:\dm\bin\codes\CodeBlocks\ProjTemp\src\DbUtilities.cpp:935)
#20 0x4023e3    test_RowChaining() (C:\dm\bin\codes\CodeBlocks\ProjTemp\main.cpp:234)
#21 0x4081c6    main() (C:\dm\bin\codes\CodeBlocks\ProjTemp\main.cpp:1034)

And here is the call stack when I step to the next line and debugger breaks one last time before resuming normal execution:

#0 0x770186ff   ntdll!DbgBreakPoint() (C:\Windows\system32\ntdll.dll:??)
#1 0x77082edb   ntdll!RtlpNtMakeTemporaryKey() (C:\Windows\system32\ntdll.dll:??)
#2 0x77052c7f   ntdll!RtlQueryRegistryValues() (C:\Windows\system32\ntdll.dll:??)
#3 0x77083f3b   ntdll!RtlpNtMakeTemporaryKey() (C:\Windows\system32\ntdll.dll:??)
#4 0x7704bcfd   ntdll!EtwSendNotification() (C:\Windows\system32\ntdll.dll:??)
#5 0x770374d5   ntdll!RtlEnumerateGenericTableWithoutSplaying() (C:\Windows\system32\ntdll.dll:??)
#6 0x75829dc6   KERNEL32!HeapFree() (C:\Windows\system32\kernel32.dll:??)
#7 0x75a99c03   msvcrt!free() (C:\Windows\system32\msvcrt.dll:??)
#8 0x350000 ?? () (??:??)
--> #9 0x534020 free_wrapper(pv=0x352af0) (C:\dm\bin\codes\CodeBlocks\ProjTemp\src\Unrelated\MemMgmt.cpp:282)
#10 0x407f74    operator delete(pv=0x352af0) (C:\dm\bin\codes\CodeBlocks\ProjTemp\main.cpp:1002)
#11 0x629a74    __gnu_cxx::new_allocator<char>::deallocate(this=0x22f718, __p=0x352af0 "\nÿÿÿÿÿÿº\r%") (C:/Program Files/CodeBlocks/MinGW/lib/gcc/mingw32/4.9.2/include/c++/ext/new_allocator.h:110)
#12 0x6c2257    std::allocator_traits<std::allocator<char> >::deallocate(__a=..., __p=0x352af0 "\nÿÿÿÿÿÿº\r%", __n=50) (C:/Program Files/CodeBlocks/MinGW/lib/gcc/mingw32/4.9.2/include/c++/bits/alloc_traits.h:383)
#13 0x611940    basic_CDataUnit<std::allocator<char> >::~basic_CDataUnit(this=0x22f714, __vtt_parm=0x781df4 <VTT for basic_CDataUnit_TDB<std::allocator<char> >+4>, __in_chrg=<optimized out>) (include/DataUnit/CDataUnit.h:112)
#14 0x61dfa1    basic_CDataUnit_TDB<std::allocator<char> >::~basic_CDataUnit_TDB(this=0x22f714, __in_chrg=<optimized out>, __vtt_parm=<optimized out>) (include/DataUnit/CDataUnit_TDB.h:125)
#15 0x503898    CTblSegHandle::UpdateChainedRowData(this=0x353cf8, new_row_data=..., old_row_fetch_res=..., vColTypes=..., block_hnd=...) (C:\dm\bin\codes\CodeBlocks\ProjTemp\src\SegHandles\CTblSegHandle.cpp:912)
#16 0x502fcc    CTblSegHandle::UpdateRowData(this=0x353cf8, new_row_data=..., old_row_fetch_res=..., vColTypes=..., block_hnd=...) (C:\dm\bin\codes\CodeBlocks\ProjTemp\src\SegHandles\CTblSegHandle.cpp:764)
#17 0x443272    UpdateRow(row_addr=..., new_data_unit=..., vColTypes=..., block_hnd=..., seg_hnd=...) (C:\dm\bin\codes\CodeBlocks\ProjTemp\src\DbUtilities.cpp:910)
#18 0x443470    UpdateRow(row_addr=..., vColValues=..., vColTypes=...) (C:\dm\bin\codes\CodeBlocks\ProjTemp\src\DbUtilities.cpp:935)
#19 0x4023e3    test_RowChaining() (C:\dm\bin\codes\CodeBlocks\ProjTemp\main.cpp:234)
#20 0x4081c6    main() (C:\dm\bin\codes\CodeBlocks\ProjTemp\main.cpp:1034)
Kemal
  • 849
  • 5
  • 21
  • Free only has undefined behaviour, if the memory has already been free'd. Check your pointers and maybe try to use delete (if you have all the used memory in one object). You can also check the pointer you give to the function before (`if (ptr) free(ptr); else ...`) – R. Joiny Jul 18 '17 at 09:15
  • @R.Joiny : I am not sure what you mean exactly but you can assume all `ptr`s passed to `free()` are non-null and already allocated by `malloc()` due to the fact that I am using custom `::operator new` and `::operator delete` which check for those situations. – Kemal Jul 18 '17 at 09:26
  • 2
    Looks like you scrambled up your heap or call delete for an already deleted object. Kinda hard to find thing. Remove more'n'more from your code until the error vanished, or set all pointer to object to 0 until your deleted the corresponding pointer. Then you use special delete/free function that checks for non-0 argument. That might help, but if you make copy of a pointer you wil miss it. – harper Jul 18 '17 at 11:13
  • @harper : The custom `::operator new` keeps records of current allocations(pointer and size), and custom `::operator delete` uses those records to check if the memory is **actually** allocated before freeing it. So double release is out of the question as far as I am concerned. – Kemal Jul 18 '17 at 11:37
  • With that said, point of crash turns out **not** to be consistent in **non-debug** mode This is contrary to what I wrote in my OP (correction is due shortly). So it seems to me now to be **some** case of buffer overflow **somewhere**. Looks like I'm in for a nightmare. – Kemal Jul 18 '17 at 11:44
  • VS Community has some very good graphical memory debugging tools. Maybe you can check them out if it helps :) – R. Joiny Jul 18 '17 at 11:46
  • Maybe the memory that you use was reallocated, for example, but you saved pointer to this memory and is still using it(pointer that points to released memory). Just for thought. – Hardwired Jul 18 '17 at 12:09
  • 1
    This kind of bug report typically means one of three things: 1) a double free, 2) freeing a pointer that was never actually allocated, or 3) the heap has been corrupted. Take a look at the memory being pointed to b y the pointer you're trying to free; if it contains one of the 'signature' values mention in the following link you might get a clue about what's going wrong: https://stackoverflow.com/a/370362/12711 – Michael Burr Jul 18 '17 at 17:34
  • 2
    And even though you're using GCC, since it's relying on the `msvcrt` DLL the following article might be helpful: https://learn.microsoft.com/en-us/visualstudio/debugger/crt-debug-heap-details – Michael Burr Jul 18 '17 at 17:38
  • 2
    And you might use` gflags` to set each heap allocation on it's own page which might help trigger the debugger to get notified when the corruption occurs instead of detecting it after the fact: https://learn.microsoft.com/en-us/windows-hardware/drivers/debugger/gflags-and-pageheap – Michael Burr Jul 18 '17 at 18:09
  • @R.Joiny: "Free **only** has undefined behaviour, if the memory has already been free'd" is untrue. It also has undefined behavior if you pass a pointer to a global object, a stack object, or an object not at the start of a `malloc'ed` block. – MSalters Jul 19 '17 at 06:44
  • 1
    @MichaelBurr : I found the bug. Buffer overflow as expected. I had never used Visual Studio before so it actually took me longer to get it up and running with my project than finding the bug. But without it, it would certainly be much more troublesome. Reported the heap corruption right away. From there on, it was just a couple of minutes of signiture memory value inspection, as you suggested. Thank you for the advice. – Kemal Jul 20 '17 at 11:24

2 Answers2

4

When I see a call stack that looks like yours the most common cause is heap corruption. A double free or attempting to free a pointer that was never allocated can have similar call stacks. Since you characterize the crash as inconsistent that makes heap corruption the more likely candidate. Double frees and freeing unallocated pointers tend to crash consistently in the same place. To hunt down issues like this I usually:

  1. Install Debugging Tools for Windows
  2. Open a command prompt with elevated privileges
  3. Change directory to the directory that Debugging Tools for Windows is installed in.
  4. Enable full page heap by running gflags.exe -p /enable applicationName.exe /full
  5. Launch application with debugger attached and recreate the issue.
  6. Disable full page heap for the application by running gflags.exe -p /disable applicationName.exe

Running the application with full page heap places an inaccessible page at the end of each allocation so that the program stops immediately if it accesses memory beyond the allocation. This is according to the page GFlags and PageHeap. If a buffer overflow is causing the heap corruption this setting should cause the debugger to break when the overflow occurs..

Make sure to disable page heap when you are done debugging. Running under full page heap can greatly increase memory pressure on an application by making every heap allocation consume an entire page.

Gene Pool
  • 61
  • 5
  • Wow, this actually helped me. After enabling the flag I got a crash exactly at the spot I screwed up my memory freeing logic, instead of some random confusing place not related to the problem! – ScienceDiscoverer Aug 10 '23 at 14:38
-1

You can use valgrind to check if there is any invalid read /write or any invalid free is there in your CODE. valgrind -v --leak-check=full --show-reachable=yes --log-file=log_valgrind ./Process

log_valgrind will contains invalid read/write.

Rohit
  • 142
  • 8