3

I'm using libFuzzer to fuzz an API.
The API is deserializing an array of bits (given by libFuzzer)
and converting them into c++ class instantiations.

Due to the serialization format, libFuzer is able construct a serialized object that tell the deserializer to reserve large amounts of data (which cannot be met).
This is done through calls to std::vector::resize(). The vector throws a std::bad_alloc, and although the problem is caught and safely mitigated, it causes extreme lag in the fuzzer (as mentioned in the following documentation on OOM issues).

In an attempt to lower the amount of memory used when the fuzzer is running, I was hoping to set ulimit -v and adjust the available virtual memory of the process. However doing so causes 

==27609==ERROR: AddressSanitizer failed to allocate 0xdfff0001000 (15392894357504) bytes at address 2008fff7000 (errno: 12)
==27609==ReserveShadowMemoryRange failed while trying to map 0xdfff0001000 bytes. Perhaps you're using ulimit -v

Why can't the address sanitizer work under ulmit -v?
I wish it could, then I might be able to fuzz more effectively.

Other information:
My build flags were: 

copts = [
    "-fsanitize=address,fuzzer",
    "-fsanitize-trap=undefined,integer",
    "-fsanitize-coverage=trace-pc,trace-cmp,trace-pc-guard",
    "-g",
    "-O0",
    "-fno-omit-frame-pointer",
    "-fno-sanitize=vptr",
],
linkopts = [
    "-fsanitize=address,fuzzer",
    "-fsanitize-trap=undefined,integer",
    "-fno-sanitize=vptr",
    "-fsanitize-link-c++-runtime",
],

I tried turning flags off so I could set ulimit and run the fuzzer: 

copts = [
    "-fsanitize=fuzzer",
    "-g",
    "-O0",
    "-fno-omit-frame-pointer",
],
linkopts = [
    "-fsanitize=fuzzer",
],

but this causes an immediate segfault.

Trevor Hickey
  • 36,288
  • 32
  • 162
  • 271
  • Is it possible to see `/proc/PID/maps` before crash? I wonder if shadow memory range is occupied by someone before Asan has chance to initialize and `mmap` it. – yugr Oct 24 '18 at 14:34
  • `ulimit` doesn't make you use less memory, it just prevents you from using more memory. What are you expecting to happen if the application wants more than the limit, if not an error? – Barmar Oct 24 '18 at 21:25
  • @Barmar yeah, I was hoping that an artificial memory cap would cause the program to OOM more often, but reduce the latency lag. – Trevor Hickey Oct 25 '18 at 03:08

1 Answers1

1

Asan reserves 1/8-th of process address space for shadow memory at startup to hold status of user data (allocated, freed, etc.). This is by design and there is nothing one can do about it.

Note that you generally don't care about virtual memory but rather physical one (which is also causing new to fail in your case).

yugr
  • 19,769
  • 3
  • 51
  • 96