5

I have an issue where any Leak Sanitizer backtraces that go through dynamically loaded libraries report Unknown Module for any function calls within that library.

Direct leak of 48 byte(s) in 1 object(s) allocated from:
    #0 0x4e3e36 in malloc (/usr/sbin/radiusd+0x4e3e36)
    #1 0x7fb406e95f69  (<unknown module>)
    #2 0x7fb406eafc36  (<unknown module>)
    #3 0x7fb406eafd40  (<unknown module>)
    #4 0x7fb406ea3364  (<unknown module>)
    #5 0x7fb4063de7d4  (<unknown module>)
    #6 0x7fb4063c61c4  (<unknown module>)
    #7 0x7fb406617863  (<unknown module>)
    #8 0x7fb415620681 in dl_load_func /usr/src/debug/freeradius-server-4.0.0/src/main/dl.c:194:34
    #9 0x7fb41561edab in dl_symbol_init_walk /usr/src/debug/freeradius-server-4.0.0/src/main/dl.c:301:7
    #10 0x7fb41561df1e in dl_module /usr/src/debug/freeradius-server-4.0.0/src/main/dl.c:748:6
    #11 0x7fb41561f3db in dl_instance /usr/src/debug/freeradius-server-4.0.0/src/main/dl.c:853:20
    #12 0x7fb41564f4ab in module_bootstrap /usr/src/debug/freeradius-server-4.0.0/src/main/module.c:827:6
    #13 0x7fb41564ed56 in modules_bootstrap /usr/src/debug/freeradius-server-4.0.0/src/main/module.c:1070:14
    #14 0x5352bb in main /usr/src/debug/freeradius-server-4.0.0/src/main/radiusd.c:561:6
    #15 0x7fb41282ab34 in __libc_start_main (/lib64/libc.so.6+0x21b34)
    #16 0x4204ab in _start (/usr/sbin/radiusd+0x4204ab)

I've had an almost identical issue with valgrind before, and I know it's due to the libraries being unloaded with dlclose on exit, and the symbols being unavailable when the symbolizer runs.

With valgrind the fix is simple

/*
 *  Only dlclose() handle if we're *NOT* running under valgrind
 *  as it unloads the symbols valgrind needs.
 */
if (!RUNNING_ON_VALGRIND) dlclose(module->handle);        /* ignore any errors */

RUNNING_ON_VALGRIND being a macro provided by the valgrind library for detecting if the program is being valground.

I can't see anything in the LSAN docs for a similar feature for when ASAN_OPTIONS=detect_leaks=1 is set.

Does anyone know if it's possible to perform a runtime check for running under LSAN?

Arran Cudbard-Bell
  • 5,912
  • 2
  • 26
  • 48
  • Isn't LSAN something you compile with? I mean isn't it determined in compilation, unlike Valgrind that runs your program so only in runtime? – kabanus Apr 04 '18 at 17:58
  • Possible duplicate of [How can I know if Leak Sanitizer is enabled at compile time?](https://stackoverflow.com/questions/31273016/how-can-i-know-if-leak-sanitizer-is-enabled-at-compile-time) - this being my point. – kabanus Apr 04 '18 at 17:59
  • Yes you need to link to the ASAN library for support, but no, it's toggle-able with `ASAN_OPTIONS=detect_leaks=(1|0)` and seems to default to 0 on my system. – Arran Cudbard-Bell Apr 04 '18 at 18:00
  • Not a duplicate. That question is asking about compile time checking, this is runtime. – Arran Cudbard-Bell Apr 04 '18 at 18:01
  • I see, retracted dup vote. So you compile with it, but the system can turn it of in runtime. Sounds difficult, thanks for the clarification. – kabanus Apr 04 '18 at 18:02
  • 2
    The community tend to come up with creative solutions to these things. My favourite was one for detecting being run under GDB, where you fork and then try and ``pattach()`` to your parent. Not at all reliable, but fun! – Arran Cudbard-Bell Apr 04 '18 at 18:04
  • https://llvm.org/svn/llvm-project/compiler-rt/trunk/include/sanitizer/lsan_interface.h This looks potentially promising `__lsan_is_turned_off`` ... I wonder if that's only called if LSAN is enabled... – Arran Cudbard-Bell Apr 04 '18 at 18:11
  • Is Valgrind workaround common? if yes, could you mention it in [Asan tracker](https://github.com/google/sanitizers/issues/89) for other poor souls who suffer from this? – yugr Apr 04 '18 at 18:21
  • The approach is - not calling dlclose(). It's mentioned by @kcc in that GitHub issue. Commented on the issue with link pointing back here. – Arran Cudbard-Bell Apr 04 '18 at 18:42

2 Answers2

3

The LSAN interface headers allow the user to define a callback __lsan_is_turned_off to allow the program to disable the leak checker. This callback is only executed if LSAN is enabled.

#include <sanitizer/lsan_interface.h>

static bool running_under_lsan = false;

int __attribute__((used)) __lsan_is_turned_off(void)
{
    running_under_lsan = true;
    return 0;
}

EDIT: It's actually more complicated than that. As @yugr commented It appears __lsan_is_turned_off is only executed when a process or child process exits.

There is however a solution!

#include <stdio.h>
#include <stdlib.h>
#include <stdbool.h>
#include <unistd.h>

#include <string.h>
#include <errno.h>

#include <sanitizer/common_interface_defs.h>

static int from_child[2] = {-1, -1};
static int pid;

int __attribute__((used)) __lsan_is_turned_off(void)
{
    uint8_t ret = 1;

    /* Parent */
    if (pid != 0) return 0;

    /* Child */
    if (write(from_child[1], &ret, sizeof(ret)) < 0) {
        fprintf(stderr, "Writing LSAN status failed: %s", strerror(errno));
    }
    close(from_child[1]);
    return 0;
}

int main(int argc, char **argv)
{
    uint8_t ret = 0;

    if (pipe(from_child) < 0) {
        fprintf(stderr, "Failed opening internal pipe: %s", strerror(errno));
        exit(EXIT_FAILURE);
    }

    pid = fork();
    if (pid == -1) {
        fprintf(stderr, "Error forking: %s", strerror(errno));
        exit(EXIT_FAILURE);
    }

    /* Child */
    if (pid == 0) {
        close(from_child[0]);   /* Close parent's side */
        exit(EXIT_SUCCESS);
    }

    /* Parent */
    close(from_child[1]);       /* Close child's side */

    while ((read(from_child[0], &ret, sizeof(ret)) < 0) && (errno == EINTR));

    close(from_child[0]);       /* Close our side (so we don't leak FDs) */

    /* Collect child */
    waitpid(pid, NULL, 0);

    if (ret) {
        printf("Running under LSAN\n");
    } else {
        printf("Not running under LSAN\n");
    }

    exit(EXIT_SUCCESS);
}

Example:

clang -g3 -fsanitize=address foo.c

ASAN_OPTIONS='detect_leaks=1' ./a.out
Running under LSAN

ASAN_OPTIONS='detect_leaks=0' ./a.out
Not running under LSAN
Arran Cudbard-Bell
  • 5,912
  • 2
  • 26
  • 48
2

First of all, not printing stacktraces on dlclose (or printing incorrect ones) is a known issue in all sanitizers (not just LSan).

Secondly, as of now there's no API to detect that LeakSanitizer is enabled at runtime so your best bet is to manually check that program is linked against Lsan and detect_leaks=0 isn't set in environment:

void (*__lsan_is_turned_off)() = dlsym(RTLD_DEFAULT, "__lsan_is_turned_off");
const char *lsan_opts = getenv("LSAN_OPTIONS");
const char *asan_opts = getenv("ASAN_OPTIONS");
int disable_dlclose = __lsan_is_turned_off != 0 && !__lsan_is_turned_off()
  && !(lsan_opts && (strstr(lsan_opts, "detect_leaks=0") || strstr(lsan_opts, "detect_leaks=false"))
  && !(asan_opts && (strstr(asan_opts, "detect_leaks=0") || strstr(asan_opts, "detect_leaks=false"));

(__lsan_is_turned_off is defined in sanitizer/lsan_interface.h).

If you enable LSan via -fsanitize=address, you can replace __lsan_is_turned_off check with #ifdef __SANITIZE_ADDRESS__.

yugr
  • 19,769
  • 3
  • 51
  • 96
  • Ahh, nice. I'm not sure which of our methods is more reliable, but I think Stack Overflow etiquette dictates I should accept your answer :) – Arran Cudbard-Bell Apr 04 '18 at 18:22
  • Both solutions have their own flaws :) Mine does not discern between `ASAN_OPTIONS=disable_leaks=0` and `ASAN_OPTIONS=disable_leaks=1` (so will fire even if Lsan is linked but disabled at runtime) and yours will detect Lsan too late (I believe `__lsan_is_turned_off` is called only at exit). – yugr Apr 04 '18 at 18:32
  • 1
    Hmm no the callback is called on process start, process exit... and at a random point in between (not `fork()` not `pthread_create()`, not sure why). It'd need to be called on start right? To actually enable/disable the memory tracking. – Arran Cudbard-Bell Apr 04 '18 at 18:39
  • @ArranCudbard-Bell Weird, I can only see it being called atexit in source code (through `DoLeakCheck`) but I'm probly too tired. If it's indeed called at startup (which makes sense), it might be a better solution even though `lsan_interface.h` does not promise this so it's technically an implementation detail (as well as default implementation of `__lsan_is_turned_off` not looking at `detect_leaks` environment setting). So I'll have no problem to removing answer status from my reply. – yugr Apr 04 '18 at 18:49
  • You're right. I wrote a simple program and it was only called on exit. My guess is in my complex program there's some forking action going on. Whilst the callback isn't executed on fork() it is executed when the child exits. See the horror below. – Arran Cudbard-Bell Apr 04 '18 at 22:00