Why does calling the C abort() function from an x86_64 assembly function lead to segmentation fault (SIGSEGV) instead of an abort signal?

Question

Consider the program:

main.c

#include <stdlib.h>

void my_asm_func(void);
__asm__(
    ".global my_asm_func;"
    "my_asm_func:;"
    "call abort;"
    "ret;"
);

int main(int argc, char **argv) {
    if (argv[1][0] == '0') {
        abort();
    } else if (argv[1][0] == '1') {
        __asm__("call abort");
    } else {
        my_asm_func();
    }
}

Which I compile as:

gcc -ggdb3 -O0 -o main.out main.c

Then I have:

$ ./main.out 0; echo $?
Aborted (core dumped)
134
$ ./main.out 1; echo $?
Aborted (core dumped)
134
$ ./main.out 2; echo $?
Segmentation fault (core dumped)
139

Why do I get the segmentation fault only for the last run, and not an abort signal as expected?

man 7 signal:

   SIGABRT       6       Core    Abort signal from abort(3)
   SIGSEGV      11       Core    Invalid memory reference

confirms the signals due to the 128 + SIGNUM rule.

As a sanity check I also tried to make other function calls from assembly as in:

#include <stdlib.h>

void my_asm_func(void);
__asm__(
    ".global my_asm_func;"
    "my_asm_func:;"
    "lea puts_message(%rip), %rdi;"
    "call puts;"
    "ret;"
    "puts_message: .asciz \"hello puts\""
);

int main(void) {
    my_asm_func();
}

and that did work and print:

hello puts

Tested in Ubuntu 19.04 amd64, GCC 8.3.0, glibc 2.29.

I also tried it in an Ubunt Ubuntu 18.04 docker, and the results were the same, except that the program outputs when running:

./main.out: Symbol `abort' causes overflow in R_X86_64_PC32 relocation          
./main.out: Symbol `abort' causes overflow in R_X86_64_PC32 relocation

which feels like a good clue.

Answer 1

In this code that defines a function at global scope (with basic assembly):

void my_asm_func(void);

__asm__(
    ".global my_asm_func;"
    "my_asm_func:;"
    "call abort;"
    "ret;"
);

You violate one of the x86-64(AMD64) System V ABI rules that requires 16 byte stack alignment (may be higher depending on the parameters) at a point just before a CALL is made.

3.2.2 The Stack Frame

In addition to registers, each function has a frame on the run-time stack. This stack grows downwards from high addresses. Figure 3.3 shows the stack organization.

The end of the input argument area shall be aligned on a 16 (32, if __m256 is passed on stack) byte boundary. In other words, the value (%rsp + 8) is always a multiple of 16 (32) when control is transferred to the function entry point. The stack pointer, %rsp, always points to the end of the latest allocated stack frame.

Upon entry to a function the stack will be misaligned by 8 because the 8 byte return address is now on the stack. To align the stack back on a 16 byte boundary subtract 8 from RSP at the beginning of the function and add 8 back to RSP when finished. You can also just push any register like RBP at the beginning and pop it after to get the same effect.

This version of the code should work:

void my_asm_func(void);

__asm__(
    ".global my_asm_func;"
    "my_asm_func:;"
    "push %rbp;"
    "call abort;"
    "pop %rbp;"
    "ret;"
);

---

Regarding this code that happened to work:

__asm__("call abort");

The compiler likely generated the main function in such away that the stack was aligned on a 16 byte boundary prior to this call so it happened to work. You shouldn't rely on this behavior. There are other potential issues with this code, but don't present as a failure in this case. The stack should be properly aligned before the call; you should be concerned in general about the red zone; and you should specify all the volatile registers in the calling conventions as clobbers including RAX/RCX/RDX/R8/R9/R10/R11, the FPU registers, and the SIMD registers. In this case abort never returns so this isn't an issue related to your code.

The red-zone is defined in the ABI this way:

The 128-byte area beyond the location pointed to by %rsp is considered to be reserved and shall not be modified by signal or interrupt handlers.8 Therefore, functions may use this area for temporary data that is not needed across function calls. In particular, leaf functions may use this area for their entire stack frame, rather than adjusting the stack pointer in the prologue and epilogue. This area is known as the red zone.

It is generally a bad idea to call a function in inline assembly. An example of calling printf can be found in this other Stackoverflow answer which shows the complexities of doing a CALL especially in 64-bit code with red-zone. David Wohlferd's Dont Use Inline Asm is always a good read.

---

This code happened to work:

void my_asm_func(void);
__asm__(
    ".global my_asm_func;"
    "my_asm_func:;"
    "lea puts_message(%rip), %rdi;"
    "call puts;"
    "ret;"
    "puts_message: .asciz \"hello puts\""
);

but you were probably lucky that puts didn't need proper alignment and you happened to get no failure. You should be aligning the stack before calling puts as described earlier with the my_asm_func that called abort. Ensuring compliance with the ABI is the key to ensuring code will work as expected.

---

Regarding the relocation errors, that is probably because the version of Ubuntu being used is using Position Independent Code (PIC) by default for GCC code generation. You could fix the issue by making the C library calls though the Procedure Linkage Table by appending @plt to the function names you CALL. Peter Cordes wrote a related Stackoverflow answer on this topic.