Calling dlsym from a function other than _start crashes

Question

I have this assembly program that works fine:

SECTION .text

EXTERN dlopen ; loads a dynamic library
EXTERN dlsym ; retrieves the address for a symbol in the dynamic library

global _start ; "global" means that the symbol can be accessed in other modules. In order to refer to a global symbol from another module, you must use the "extern" keyboard
_start:

    ; load the library
    mov rdi, str_libX11so
    mov rsi, 2; RTLD_NOW=2
    call dlopen wrt ..plt
        ; PLT stands for Procedure Linkage Table:
        ; used to call external library functions whose address is not know at link time,
        ; so it must be resolved by the dynamic linker at run time
        ; more info: https://reverseengineering.stackexchange.com/questions/1992/what-is-plt-got
    mov [ptr_libX11so], rax ; the previous function call returned the value in rax

    ; load the function
    mov rdi, [ptr_libX11so]
    mov rsi, fstr_XOpenDisplay
    call dlsym wrt ..plt
    mov [fptr_XOpenDisplay], rax

    mov rax, 60 ; syscal: exit
    mov rdi, 0 ; return code
    syscall

str_libX11so: db "libX11.so", 0

; X11 function names
fstr_XOpenDisplay: db "XOpenDisplay", 0


SECTION .data
ptr_libX11so: dq 0 ; ptr to the X11 library

; X11 function ptrs
fptr_XOpenDisplay: dq 0

Then I tried to move the code that calls dlsym into a function (loadX11Functions).

SECTION .text

EXTERN dlopen ; loads a dynamic library
EXTERN dlsym ; retrieves the address for a symbol in the dynamic library

loadX11Functions:
    mov rdi, [ptr_libX11so]
    mov rsi, fstr_XOpenDisplay
    call dlsym wrt ..plt
    mov [fptr_XOpenDisplay], rax
    ret

global _start ; "global" means that the symbol can be accessed in other modules. In order to refer to a global symbol from another module, you must use the "extern" keyboard
_start:

    ; load the library
    mov rdi, str_libX11so
    mov rsi, 2; RTLD_NOW=2
    call dlopen wrt ..plt
        ; PLT stands for Procedure Linkage Table:
        ; used to call external library functions whose address is not know at link time,
        ; so it must be resolved by the dynamic linker at run time
        ; more info: https://reverseengineering.stackexchange.com/questions/1992/what-is-plt-got
    mov [ptr_libX11so], rax ; the previous function call returned the value in rax

    call loadX11Functions

    mov rax, 60 ; syscal: exit
    mov rdi, 0 ; return code
    syscall

str_libX11so: db "libX11.so", 0

; X11 function names
fstr_XOpenDisplay: db "XOpenDisplay", 0


SECTION .data
ptr_libX11so: dq 0 ; ptr to the X11 library

; X11 function ptrs
fptr_XOpenDisplay: dq 0

I'm confused that such a small change breaks my program.

This is the command I use for compiling:

nasm -f elf64 -g -F dwarf minimal.asm && gcc -nostartfiles -no-pie minimal.o -ldl -o minimal && ./minimal

Probably stack alignment. RSP is aligned by 16 on entry to _start (guaranteed by the ABI), but your functions don't preserve that alignment before calling library functions. — Peter Cordes, Jan 19 '21 at 22:32
@PeterCordes Thanks! I didn't know that ABI restriction. I've just added `sub rsp, 8` at the beginning of the function and `add rsp, 8` at the end of the function. It seems to work! — tuket, Jan 19 '21 at 22:41
Yes, that confirms stack alignment was the problem. And yes, that's the correct place to put sub/add rsp instructions. Some compilers will use a "dummy" push/pop because it can actually be cheaper than `sub rsp` (code size, and even uops because of the stack engine), but GCC uses sub/add with the default tuning so there's nothing wrong with that. In fact, if you'd looked at GCC's output with `gcc -O3` but using attributes or separate files to prevent inlining, that's what you'd see. (https://godbolt.org/) — Peter Cordes, Jan 19 '21 at 22:45
@PeterCordes Is that macro a trick so the compiler doesn't inline the function? I see that `and rsp,0xfffffffffffffff0` makes the stack pointer 16-bit aligned by subtracting, but I don't see how that gets recovered. And right after that it pushes `rax` and `rsp`, I don't understand why :s — tuket, Jan 20 '21 at 20:30
What macro? I didn't mention or link to anything that used `and` to align RSP from unknown incoming stack alignment. Or anything with asm or CPP macros. (And BTW, using `and` to align the stack if you don't know the incoming alignment would mean you need to save the old RSP, e.g. in RBP, so you can restore it exactly. That's why we normally make functions that assume 16-byte stack alignment before the call, and do a odd number of 8-byte adjustments to RSP before another call.) — Peter Cordes, Jan 20 '21 at 20:32
@PeterCordes This macro `#define macro(A)inline A { printf("hello\n"); }` from the godbolt link that you sent :) — tuket, Jan 20 '21 at 20:35
Godbolt "remembers" whatever code you last had open in it. Check the URL, my Godbolt link was just a bare top-level one to `https://godbolt.org/`, not to any code on it. IDK what the purpose of that macro is in whatever you were looking at. Looks like it's to define a simple inline function with your choice of name. — Peter Cordes, Jan 20 '21 at 20:38

Calling dlsym from a function other than _start crashes

0 Answers0