14

That's what I understood by reading some memory segmentation documents: when a function is called, there are a few instructions (called function prologue) that save the frame pointer on the stack, copy the value of the stack pointer into the base pointer and save some memory for local variables.

Here's a trivial code I am trying to debug using GDB:

void test_function(int a, int b, int c, int d) {
    int flag;
    char buffer[10];

    flag = 31337;
    buffer[0] = 'A';
}

int main() {
    test_function(1, 2, 3, 4);
}

The purpose of debugging this code was to understand what happens in the stack when a function is called: so I had to examine the memory at various step of the execution of the program (before calling the function and during its execution). Although I managed to see things like the return address and the saved frame pointer by examining the base pointer, I really can't understand what I'm going to write after the disassembled code.

Disassembling:

(gdb) disassemble main
Dump of assembler code for function main:
   0x0000000000400509 <+0>: push   rbp
   0x000000000040050a <+1>: mov    rbp,rsp
   0x000000000040050d <+4>: mov    ecx,0x4
   0x0000000000400512 <+9>: mov    edx,0x3
   0x0000000000400517 <+14>:    mov    esi,0x2
   0x000000000040051c <+19>:    mov    edi,0x1
   0x0000000000400521 <+24>:    call   0x4004ec <test_function>
   0x0000000000400526 <+29>:    pop    rbp
   0x0000000000400527 <+30>:    ret    
End of assembler dump.
(gdb) disassemble test_function 
Dump of assembler code for function test_function:
   0x00000000004004ec <+0>: push   rbp
   0x00000000004004ed <+1>: mov    rbp,rsp
   0x00000000004004f0 <+4>: mov    DWORD PTR [rbp-0x14],edi
   0x00000000004004f3 <+7>: mov    DWORD PTR [rbp-0x18],esi
   0x00000000004004f6 <+10>:    mov    DWORD PTR [rbp-0x1c],edx
   0x00000000004004f9 <+13>:    mov    DWORD PTR [rbp-0x20],ecx
   0x00000000004004fc <+16>:    mov    DWORD PTR [rbp-0x4],0x7a69
   0x0000000000400503 <+23>:    mov    BYTE PTR [rbp-0x10],0x41
   0x0000000000400507 <+27>:    pop    rbp
   0x0000000000400508 <+28>:    ret    
End of assembler dump.

I understand that "saving the frame pointer on the stack" is done by " push rbp", "copying the value of the stack pointer into the base pointer" is done by "mov rbp, rsp" but what is getting me confused is the lack of a "sub rsp $n_bytes" for "saving some memory for local variables". I've seen that in a lot of exhibits (even in some topics here on stackoverflow).

I also read that arguments should have a positive offset from the base pointer (after it's filled with the stack pointer value), since if they are located in the caller function and the stack grows toward lower addresses it makes perfect sense that when the base pointer is updated with the stack pointer value the compiler goes back in the stack by adding some positive numbers. But my code seems to store them in a negative offset, just like local variables.. I also can't understand why they are put in those registers (in the main).. shouldn't they be saved directly in the rsp "offsetted"?

Maybe these differences are due to the fact that I'm using a 64 bit system, but my researches didn't lead me to anything that would explain what I am facing.

Peter Cordes
  • 328,167
  • 45
  • 605
  • 847
Mark
  • 405
  • 4
  • 10
  • 3
    can you add a call to another function in your test_function? It looks like rsp is not updated, because it is not used in your function. If you call another function, it must be updated. – wimh Feb 24 '15 at 10:59
  • yeah, I made another trivial function called "second_test()" with just printed a string.. now in the test_function I have the sub instruction for the rsp! thanks – Mark Feb 24 '15 at 12:08
  • Another duplicate: [Compiler using local variables without adjusting RSP](https://stackoverflow.com/q/43013693) – Peter Cordes Nov 09 '22 at 21:27

2 Answers2

20

The System V ABI for x86-64 specifies a red zone of 128 bytes below %rsp. These 128 bytes belong to the function as long as it doesn't call any other function (it is a leaf function).

Signal handlers (and functions called by a debugger) need to respect the red zone, since they are effectively involuntary function calls.
All of the local variables of your test_function, which is a leaf function, fit into the red zone, thus no adjustment of %rsp is needed. (Also, the function has no visible side-effects and would be optimized out on any reasonable optimization setting).

You can compile with -mno-red-zone to stop the compiler from using space below the stack pointer. Kernel code has to do this because hardware interrupts don't implement a red-zone.

Peter Cordes
  • 328,167
  • 45
  • 605
  • 847
EOF
  • 6,273
  • 2
  • 26
  • 50
  • 2
    Interrupt handlers run on the kernel stack. *Signal* handlers would be a better example of asynchronous "your process is temporarily running some other code" events. – Peter Cordes Jul 14 '15 at 04:25
  • 1
    @PeterCordes: citing `AMD64 ABI Draft 1.0, Section 3.2.2 The Stack Frame: [...]The 128-byte area beyond the location pointed to by %rsp is considered to be reserved and shall not be modified by signal or interrupt handlers[...]`. Note the "or **interrupt handlers**". – EOF Jul 15 '15 at 23:36
  • Ok, but the way int handlers avoid the modification on any system that protects the kernel from user-space is by having a separate kernel stack. Otherwise you just modify the stack from another thread during a system call or other ISR -> pwned. Or syscall with `%rsp = NULL` -> crash. http://stackoverflow.com/a/12912556/224132 for more. So this part of the ABI is only worth mentioning for OSes that don't use virtual memory, or that don't do priv sep. Or I guess it's relevant for kernel threads, but not user threads. – Peter Cordes Jul 16 '15 at 02:11
  • 2
    Update: [Kernel code can't use a red-zone](https://stackoverflow.com/questions/25787408/why-cant-kernel-code-use-a-red-zone/38043511#38043511) specifically because x86-64 interrupt handling can't respect a red-zone. You can only have a red-zone on a stack that isn't used for hardware interrupts. So that phrasing in the ABI is telling us that user stacks aren't used for interrupts at all. But on a multi-tasking OS we already know that's true for security reasons, like in 32-bit Linux or even Windows. Only at most software-driven async events like signals or GDB `print foo()` function eval. – Peter Cordes Jul 24 '19 at 15:20
  • (update 2: mevets posted an answer on that kernel red-zone Q&A pointing out that with appropriate TSS setup, you can have interrupts use an alternate stack even in kernel mode, allowing a red-zone there. So the ABI doc's phrasing including interrupt handlers may have been intentional with that in mind. Or not, hard to know.) – Peter Cordes Aug 28 '23 at 16:56
3

But my code seems to store them in a negative offset, just like local variables

The first x86_64 arguments are passed on registers, not on the stack. So when rbp is set to rsp, they are not on the stack, and cannot be on a positive offset.

They are being pushed only to:

  • save register state for a second function call.

    In this case, this is not required since it is a leaf function.

  • make register allocation easier.

    But an optimized allocator could do a better job without memory spill here.

The situation would be different if you had:

  • x86_64 function with lots of arguments. Those that don't fit on registers go on the stack.
  • IA-32, where every argument goes on the stack.

the lack of a "sub rsp $n_bytes" for "saving some memory for local variables".

The missing sub rsp on red zone of leaf function part of the question had already been asked at: Why does the x86-64 GCC function prologue allocate less stack than the local variables?

Community
  • 1
  • 1
Ciro Santilli OurBigBook.com
  • 347,512
  • 102
  • 1,199
  • 985