53

I want some tool to diagnose use-after-free bugs and uninitialized bugs. I am considering Sanitizer(Memory and/or Address) and Valgrind. But I have very little idea about their advantages and disadvantages. Can anyone tell the main features, differences and pros/cons of Sanitizer and Valgrind?

Edit: I found some of comparisons like: Valgrind uses DBI(dynamic binary instrumentation) and Sanitizer uses CTI(compile-time instrumentation). Valgrind makes the program much slower(20x) whether Sanitizer runs much faster than Valgrind(2x). If anyone can give me some more important points to consider, it will be a great help.

ch271828n
  • 15,854
  • 5
  • 53
  • 88
kayas
  • 703
  • 1
  • 5
  • 14

2 Answers2

58

I think you'll find this wiki useful.

TLDR main advantages of sanitizers are

  • much smaller CPU overheads (Lsan is practically free, UBsan/Isan is 1.25x, Asan and Msan are 2-4x for computationally intensive tasks and 1.05-1.1x for GUIs, Tsan is 5-15x)
  • wider class of detected errors (stack and global overflows, use-after-return/scope)
  • full support of multi-threaded apps (Valgrind support for multi-threading is a joke)
  • much smaller memory overhead (up to 2x for Asan, up to 3x for Msan, up to 10x for Tsan which is way better than Valgrind)

Disadvantages are

  • more complicated integration (you need to teach your build system to understand Asan and sometimes work around limitations/bugs in Asan itself, you also need to use relatively recent compiler)
  • MemorySanitizer is not reall^W easily usable at the moment as it requires one to rebuild all dependencies under Msan (including all standard libraries e.g. libc++); this means that casual users can only use Valgrind for detecting uninitialized errors
  • sanitizers typically can not be combined with each other (the only supported combination is Asan+UBsan+Lsan) which means that you'll have to do separate QA runs to catch all types of bugs
yugr
  • 19,769
  • 3
  • 51
  • 96
  • 5
    Note that valgrind supports multi-threaded apps properly. However, only one thread executes at the same time (so effectively, an application running under valgrind runs like if it was on a single core) – phd Nov 13 '17 at 22:17
  • 1
    @phd Yeah, that's what I meant, they are effectively single-threaded. In many cases this is a performance killer. – yugr Nov 14 '17 at 09:47
  • About the thread safety in Sanitizer, I found ThreadSanitizer : https://clang.llvm.org/docs/ThreadSanitizer.html . Is it good enough to handle multi-threaded execution? Can I use ThreadSanitizer with other sanitizers like: MemorySanitizer? – kayas Nov 14 '17 at 17:18
  • @kayas Regarding TSan - yes, all sanitizers fully support multi-threading. Note however that TSan's overhead is much higher (8x CPU) and it only works on x86_64 Linux. – yugr Nov 14 '17 at 17:23
  • @kayas Regarding combining sanitizers - the only combination that's supported is ASan+UBSan+LSan. Usually people run their software under each sanitizer independently. This is important disadvantage, I'll add it to the list. – yugr Nov 14 '17 at 17:24
  • You may want to move "overhead point" to advantages, since we're comparing to Valgrind. Meanwhile, TIL a disadvantage of leak sanitizer that you might add to your answer. Valgrind tracks pointers to memory, leak sanitizer doesn't. If you SIGTERM/SIGINT debuggee, [sanitizer won't print leaks](https://stackoverflow.com/q/51963570/2388257#comment113057979_51963570). Valgrind however will do so and the fact that upon signal a bunch of memory won't get freed won't clutter the report because Valgrind by default won't consider as a leak a memory to which a pointer is found. – Hi-Angel Sep 17 '20 at 11:39
  • @Hi-Angel Thanks, I moved the overhead point, it indeed makes more sense as advantage. "Valgrind tracks pointers to memory, leak sanitizer doesn't" - why do you think so? I'm adamant that Lsan does track pointers and uses it to not report leaks for reachable pointers. – yugr Sep 17 '20 at 16:43
  • @yugr hmm, weird… For some reason I can't reproduce anymore Valgrind not reporting memory leak when there's a pointer remains… Have I imagined it…? Anyway, to answer your question: [this answer demonstrates Lsan reporting memory leaked despite a pointer still in scope](https://stackoverflow.com/a/51964939/2388257). You'd probably say that memory indeed leaked there: well, to make more obvious that no pointer tracking happens here append right after the `while (argc - 1);` line a `free(a)` call. Now the app no longer leaks and you gonna get a false-positive. – Hi-Angel Sep 17 '20 at 18:30
  • @Hi-Angel The function at this link returns when signal arrives so `malloc`ed pointer remains in upper frame and is not considered to be reachable when Lsan analysis is triggered (after return from `main`). So I'd say Lsan behaves correctly in this case. – yugr Sep 18 '20 at 15:51
  • thread sanitizer works also on Mac OSX as of clang 12.0.0 https://clang.llvm.org/docs/ThreadSanitizer.html – Leo Feb 01 '21 at 20:54
  • @Leo right but the question target memory-related sanitizers. – yugr Feb 02 '21 at 04:25
3

One big difference is that the LLVM-included memory and thread sanitizers implicitly map huge swathes of address space (e.g., by calling mmap(X, Y, 0, MAP_NORESERVE|MAP_ANONYMOUS|MAP_FIXED|MAP_PRIVATE, -1, 0) across terabytes of address space in the x86_64 environment). Even though they don't necessarily allocate that memory, the mapping can play havoc with restrictive environments (e.g., ones with reasonable settings for ulimit values).

jhfrontz
  • 1,165
  • 4
  • 19
  • 31
  • 1
    "Havoc" might be too strong. App would just abort at start with easily googleable error message suggesting user to reset `ulimit`. – yugr Jun 05 '19 at 07:35
  • @yugr changing `ulimit` is sometimes easier said than done (especially in a distributed test environment that is not under your direct control); also, there are other (perhaps pathological cases) where one wants to run the sanitizer-instrumented binary under, say, `valgrind`, where the test harness sees references to this otherwise unused space as an actual use (and thus needs to allocate accounting space for it). Not to mention that it's also disconcerting/confusing to see `top` and `ps` show a process size of `124.2t`... – jhfrontz Jun 05 '19 at 17:32
  • Well, distributed test environment will always be non-trivial for Valgrind or any other dynamic tool (yes, I've tried them all) so I'm not sure how sanitizers are worse than others. Sanitized apps can not be run under valgrind (there's some memory conflict). – yugr Jun 05 '19 at 19:44