0

The callee function may use the red zone for storing local variables without the extra overhead of modifying the stack pointer. The x86-64 ABI used by System V mandates a 128-byte red zone, which begins directly after the return address and includes the function's arguments.

As Wikipedia says, the callee function can just store the variables in the red zone without modifying the stack pointer, then, why did GCC decide not to use the red zone to store my local variables, but instead it thinks that modifying rsp to allocate 0x60 bytes (96) is necessary? 128 bytes is more than enough to store my local variables, isn't it?

void test(int x, int b)
{
    char ab[60];
    strncpy(ab, "Hello World!\n", strlen("Hello World!\n") + 1);
}

// gcc (9) -g 
0x114d <+4>:    push   rbp
0x114e <+5>:    mov    rbp, rsp
0x1151 <+8>:    sub    rsp, 0x60
...
0x1196 <+77>:   call   0x1050 <__stack_chk_fail@plt>
0x119b <+82>:   leave  
0x119c <+83>:   ret

And if the callee function may use the red zone, how do we know if the function used the red zone or not? By simply looking at if the stack pointer was incremented or not? Also, why there's no add rsp, 0x60 to free up the stack space?

Peter Cordes
  • 328,167
  • 45
  • 605
  • 847
yokiyaa J
  • 1
  • what about recursion? – 0___________ Jul 08 '22 at 13:54
  • Isn't the question answered [here](https://stackoverflow.com/questions/25787408/why-cant-kernel-code-use-a-red-zone)? – habrewning Jul 08 '22 at 13:54
  • Turn the optimizer on. https://godbolt.org/z/b13MYPeKq – zwol Jul 08 '22 at 15:10
  • There's no portable way to know whether the red zone was used or not. It is inherently a platform-specific issue and any solution relies on knowing how the platform in question works. Why are you worried about it? – Jonathan Leffler Jul 08 '22 at 15:11
  • 1
    Perhaps the possibility of `call __stack_chk_fail` makes it think it's a non-leaf function, even though that can only happen with locals dead. One way to test that would be compiling with `-fno-stack-protector`, or trying on Godbolt where it's not enabled by default. – Peter Cordes Jul 08 '22 at 15:31

0 Answers0