-3

I am working with a daemon in a linux embedded device, the daemon crash randomly while running, usually after starting some hours. I investigated the crash report(stack dump) and detected that it crashes by 2 scenarios, follow these call traces:

  • Case 1. my function -> calloc -> malloc -> realloc(crash by SIGSEGV)
  • Case 2. my function -> calloc -> malloc -> realloc -> abort ->
    raise(crash by SIGABRT)

I read this link and my issue seems because of heap corruption Why do I get a C malloc assertion failure?.

I made my own wraper version for memory allocation functions (malloc, calloc, realloc and free) to attach fences around the alloced memory and monitor them by a hash table, so I can detect buffer overflow or free twice. However it still crashes without any memory violation at my fences.

So I want to ask 2 questions:

  1. Do you have any idea to debug this kind of issue?
  2. When does malloc call realloc? I looked at malloc source code at glibc briefly and see no call to realloc.
Ikarus
  • 1,169
  • 4
  • 14
  • 28
  • 7
    If you're able to run the code on hosted Linux machine, run it under valgrind. It will tell you where you're mismanaging memory. – dbush Nov 13 '18 at 14:35
  • Can you recreate this with a small test case ? and is the code running user space ? – newbie Nov 13 '18 at 14:38
  • I tried valgrind, but the daemon runs too slow and it leads to other problems to my device's operation – Ikarus Nov 13 '18 at 14:40
  • This daemon is a network appilcation at user space, it's pretty complicated with no unit test – Ikarus Nov 13 '18 at 14:41
  • 1
    Does the code always check the result of `calloc`, `malloc` and `realloc` for allocation failure? – Ian Abbott Nov 13 '18 at 14:57
  • 1
    If `malloc` is calling `realloc`, it is probably implementing `malloc(size)` as `realloc(NULL, size)`. – Ian Abbott Nov 13 '18 at 14:59
  • 1
    You detect buffer overflow or free twice - but do you detect *free not mallocated*? – Antti Haapala -- Слава Україні Nov 13 '18 at 15:50
  • @IanAbbott My malloc wraper function always check for original malloc return value – Ikarus Nov 13 '18 at 16:43
  • @AnttiHaapala, yes, free wrapper function always check whether input pointer is allocated by looking up in a hash table – Ikarus Nov 13 '18 at 16:45
  • Is it possible that the underlying `malloc` is actually buggy, perhaps in some special circumstance such as memory being exhausted or the available space being too fragmented to service a request? – John Bollinger Nov 16 '18 at 18:34
  • I am sure this would be double free or access to unallocated memory. I remember I had similar issues and that was access to unallocated memory or double free. I was sure I am not accessing unallocated memory, but when I studied code deeper I was wrong. Make sure you're not accessing unallocated memory. How many bytes are you allocating? Are these sizes large? Are you sure you're not using old pointer passed to realloc? – shjeff Nov 18 '18 at 20:39

1 Answers1

3

My colleague found the root cause(by investigating core dump file and libc malloc's source code), there is a point that it writes to a freed memory because of a bug in a linked list's delete function => corrupted memory => calloc crash.

Acually malloc does not call realloc, malloc calls __malloc_consolidate instead. In libc's obj dump file, __malloc_consolidate''s asm code is under realloc's asm code so I though malloc calls realloc.

Ikarus
  • 1,169
  • 4
  • 14
  • 28