3

My system: Ubuntu 22.04.3 running on x86_64. GCC version 11.4.0

I've read that the System V ABI mandates the red zone. From the GCC manual:

The red zone is mandated by the x86-64 ABI, it is a 128-byte area beyond the location of the stack pointer that will not be modified by signal or interrupt handlers and therefore can be used for temporary data without adjusting the stack pointer. The flag -mno-red-zone disables this red zone.

My questions:

  1. Does the red zone still exist if I use the -mno-red-zone flag in gcc?

  2. If the red zone is "disabled" does this mean that I do not comply with the system V ABI anymore?

  3. What are the consequences of this (not complying with the System V ABI)?

Peter Cordes
  • 328,167
  • 45
  • 605
  • 847
alessio solari
  • 313
  • 1
  • 6
  • 1
    I believe that flag is for use with kernel mode code where there's no red zone. It's not the application, which determines whether there's a red zone or not - it's the system. – 500 - Internal Server Error Aug 28 '23 at 14:35
  • @500 - Internal Server Error The fact that System V ABI mandates the red zone. Does this imply that if I use the -mno-red-zone flag in gcc I'm not complying with the ABI ? What are the consequences ? – alessio solari Aug 28 '23 at 14:37
  • 4
    Using `-mno-red-zone` should prevent gcc from writing below the stack pointer. It's normally free to do so, at least in user space. This "voluntary" restriction will produce less efficient code, otherwise it should not have an impact. – teapot418 Aug 28 '23 at 14:39
  • 1
    I think that flag will make GCC assume there's no red zone. As setting up the red zone is kernel responsibility, your program will still be compliant with the SYS V ABI (just less efficient). I think the kernel has already accounted for the red zone when the signal handlers registered with `sys_rt_sigaction` are invoked. If it's not the kernel, then it's probably glibc. Anyway, `-mno-red-zone` is just for the code that could use the red zone, not for the one that set it up. – Margaret Bloom Aug 28 '23 at 14:47
  • 1
    See: https://lkml.org/lkml/2014/7/24/584 _But we build the kernel with -mno-red-zone. We do *not* follow the x86-64 ABI wrt redzoning, because we *cannot*: interrupts while in kernel mode *will* use the stack without a redzone. So that "-mno-red-zone" is not some "optional guideline"._ – Craig Estey Aug 28 '23 at 15:31
  • 2
    See: [Why do we need stack allocation when we have a red zone?](https://stackoverflow.com/q/37941779/5382650) – Craig Estey Aug 28 '23 at 15:41
  • 1
    The ABI requirement is more a directive to the compiler that it must support it. You can use `-mno-red-zone` and be compliant. It just means your _leaf_ functions will have to use `%rbp` for their stack frames instead of negative offsets from `%rsp` and be less efficient. See all the linked to and "related questions" links – Craig Estey Aug 28 '23 at 15:58
  • 1
    On a side note: If you are making an OS/kernel and linking against libgcc with `-lgcc` the code in that library may assume there is a red zone. That library would have to be rebuilt with the `-mno-red-zone` option – Michael Petch Aug 28 '23 at 18:06

1 Answers1

5

The red zone still exists even if you compile code not to use it. gcc -mno-red-zone is purely a code-gen option, similar to -fno-omit-frame-pointer, or MIPS GCC's -fno-delayed-branch to leave delay slots only filled with NOP instead of useful instructions. gcc -mno-red-zone merely avoids relying on space below RSP to keep its value, just like GCC always does in non-leaf functions.

Code compiled with -mno-red-zone is ABI-compatible with code compiled with the default -mred-zone. You can freely mix these options and link the resulting .o files together into the same executable / library, or into different libraries or whatever.

The kernel's signal-delivery code and whatever else will still leave the 128 byte red zone below RSP unmodified. It doesn't know or care that the currently executing machine code isn't going to reload that data (in this case because the compiler chose not to keep anything there.) There's no mechanism for an executable to signal to the kernel that it shouldn't leave a gap of 128 bytes below the current stack pointer when delivering signals not using an alt-stack. (Because saving that small amount of stack space once is unimportant, and not worth having any extra kernel code to make it conditional. Signal handlers don't nest deeply, if at all, so it doesn't add up to a significant amount of total unused space.)

Despite it being a -m option, it doesn't have to change the ABI (the way gcc -m32 -mregparm=3 does), just not optimize in ways that take advantage of that part of the ABI.

However, kernel code usually can't use a red-zone (unless it uses the TSS and IDT machinery to use alternate stacks for interrupts even when in kernel mode, which Linux doesn't), so must be compiled with -mno-red-zone to match that ABI. Probably that's why it's a -m option: you can think of it as telling GCC that the ABI its targeting doesn't provide a red zone. x86-64 SysV with a red-zone can run code compiled for x86-64 SysV without a red-zone, but not vice versa.

Related:

Peter Cordes
  • 328,167
  • 45
  • 605
  • 847