blocking call to `dlsym` (RTLD_NEXT)

Question

I am trying to wrap function. To do that I thought about using dlsym to get the real function location in memory after having injected my dynamic library.

I am running on MacOS Mojave (10.14.6) with Apple clang version 11.0.0 (clang-1100.0.33.8).

I tried to run a rather simple code displaying a message each time malloc was called.

So I used the code of this post : Overriding 'malloc' using the LD_PRELOAD mechanism

My test code was :

#include <stdlib.h>

int main()
{
    (void)malloc(1);
    (void)malloc(1);
    return (0);
}

I injected the dynamic library using DYLD_INSERT_LIBRARIES=./mylib.so DYLD_FORCE_FLAT_NAMESPACE=1 on OSX and LD_PRELOAD=./mylib.so on Ubuntu.

On OSX, the program is blocking on the dlsym call where as on a Ubuntu docker box, the program works fine.

EDIT:

As @R.. pointed out, the dlsym implementation on OSX is calling malloc (see sources : https://opensource.apple.com/source/cctools/cctools-667.8.0/libdyld/dlopen.c)

score 2 · Accepted Answer · answered Oct 03 '19 at 16:32

2

It's likely that calling dlsym causes a recursive call to malloc, which in turn causes a recursive call to dlsym, which deadlocks the second time because it already holds some lock. Using preload libraries to wrap malloc, rather than fully replacing it, is generally a bad idea, for this and other reasons.

You might be able to use LD_AUDIT or the equivalent functionality on OSX (if any) to print a trace of when malloc is called. Otherwise you could drop in a full replacement malloc with debug output.

answered Oct 03 '19 at 16:32

R.. GitHub STOP HELPING ICE

208,859
35
376
711

Could be, because this method works fine for the `open` function – GrandChaman Oct 03 '19 at 16:34
1

@GrandChaman: Because `dlsym` doesn't call `open`. Of course `dlsym` calling `malloc` is really bad design because it means `dlsym` can spuriously fail with out-of-memory conditions, but that's the kind of bad quality of implementation I'd expect from Apple's libc... – R.. GitHub STOP HELPING ICE Oct 03 '19 at 16:40
1

Note that `fprintf()` can also call `malloc()`, so there's another path to infinite recursion. – Andrew Henle Oct 03 '19 at 16:48

score 1 · Answer 2 · answered Oct 03 '19 at 16:46

1

You may also be falling into infinite recursion due to calling fprintf(), so using this code to interpose on malloc() is dangerous (copied from the linked question):

void *malloc(size_t size)
{
    void *p = NULL;
    fprintf(stderr, "malloc(%d) = ", size);
    p = real_malloc(size);
    fprintf(stderr, "%p\n", p);
    return p;
}

Library code such as fprintf() is likely to call malloc().

answered Oct 03 '19 at 16:46

Andrew Henle

32,625
3
24
56

1

I tried removing them, didn't work either. I narrowed down the problem to the `dlsym` call itself (using `write` for debugging) – GrandChaman Oct 03 '19 at 16:49

blocking call to `dlsym` (RTLD_NEXT)

2 Answers2