Can a stack overflow result in something other than a segmentation fault?

Question

In a compiled program (let's say C or C++, but I guess this question could extend to any non-VM-ish language with a call stack) - very often when you overflow your stack, you get a segmentation fault:

Stack overflow is [a] cause, segmentation fault is the result.

Is this always the case, though? Can a stack overflow result in other kinds of program/OS behavior?

I'm asking also about non-Linux, non-Windows OSes and non-X86 hardware. (Of course if you don't have hardware memory protection or OS support for it (e.g. MS-DOS) then there's no such thing as a segmentation fault; I'm asking about cases where you could get a segmentation fault but something else happens).

Note: Assume that other than the stack overflow, the program is valid and does not try to access arrays beyond their bounds, dereference invalid pointers, etc.

Answer 1

Yes, even on a standard OS (Linux) and standard hardware (x86).

void f(void) {
    char arr[BIG_NUMBER];
    arr[0] = 0; // stack overflow
}

Note that on x86, the stack grows down, so we are assigning to the beginning of the array to trigger the overflow. The usual disclaimers apply... the exact behavior depends on more factors than are discussed in this answer, including the particulars of your C compiler.

If the BIG_NUMBER is just barely large enough to overflow, you will run into the stack guard and get a segmentation fault. That's what the stack guard is there for, and it can be as small as a single 4 KiB page (but no smaller, and this 4 KiB size is used prior to Linux 4.12) or it can be larger (1 MiB default on Linux 4.12, see mm: large stack guard gap), but it is always some particular size.

If BIG_NUMBER is large enough, the overflow can skip over the stack guard and land on some other piece of memory, potentially memory that is valid. This may result in your program behaving incorrectly but not crashing, which is basically the worst-case scenario: we want our programs to crash when they are incorrect rather than do something unintended.

Answer 2

One thing is what happens at runtime when you overflow the stack, which can be many things. Including, but not limited to; segmentation fault, overwriting variables following whatever you overflow, causing an illegal instruction, nothing at all and much more. The "old" classic paper Smashing The Stack For Fun And Profit describes a lot of ways one can have "fun" with this stuff.

Another thing is what can happen at compile time. In both C and C++, writing beyond an array or exceeding the size of the stack is Undefined Behaviour and when a program contains UB anywhere the compiler is basically free to do whatever it wants to any part of your program. And modern compilers are becoming very aggressive in exploiting UB for optimization purposes - often by assuming that UB never happens, leading them to simply remove the code containing UB or causing a branch to always or never be taken because the alternative would cause UB. Sometimes the compiler will introduce time travel or call a function that was never called in the source code and many, many other things that can cause really confusing run-time behaviour.

See also:

What Every C Programmer Should Know About Undefined Behavior #1/3

What Every C Programmer Should Know About Undefined Behavior #2/3

What Every C Programmer Should Know About Undefined Behavior #3/3

A Guide to Undefined Behavior in C and C++, Part 1

A Guide to Undefined Behavior in C and C++, Part 2

A Guide to Undefined Behavior in C and C++, Part 3

Answer 3

Other answers have covered the PC side fairly well. I'll touch on some of the issues in the embedded world.

Embedded code does have something similar to a segfault. Code is stored in some kind of non-volatile storage (usually flash these days, but some kind of ROM or PROM in the past). Writing to this needs special operations to set it up; normal memory accesses can read from it but not write to it. In addition, embedded processors usually have large gaps in their memory maps. If the processor gets a write request for memory which is read-only, or if it gets a read or write request for an address which does not physically exist, the processor will usually throw a hardware exception. If you have a debugger connected, you can check the state of the system to find what went wrong, as with a core dump.

There's no guarantee that this will happen for a stack overflow though. The stack can be placed anywhere in RAM, and this will usually be alongside other variables. The result of stack overflow will usually be to corrupt those variables.

If your application also uses heap (dynamic allocation) then it is common to assign a section of memory where the stack begins at the bottom of that section and expands upwards, and the heap begins at the top of that section and expands downwards. Clearly this means dynamically-allocated data will be the first casualty.

If you're unlucky, you may not even notice when it happens, and then you need to work out why your code is not behaving correctly. In the most ironic case, if the data being overwritten is a pointer then you may still get a hardware exception when the pointer tries to access invalid memory - but this will be some time after the stack overflow and the natural assumption will usually be that it's a bug in your code.

Embedded code has a common pattern to deal with this, which is to "watermark" the stack by initialising every byte to a known value. Sometimes the compiler can do this; or sometimes you may need to implement it yourself in the startup code before main(). You can look back from the end of the stack to find where it is no longer set to this value, at which point you know the high-water-mark for stack usage; or if it is all incorrect then you know you have an overflow. It is common (and good practise) for embedded applications to poll this continuously as a background operation, and to be able to report it for diagnostic purposes.

Having made it possible to track stack usage, most companies will set an acceptable worst-case margin to avoid overflows. This is typically somewhere from 75% to 90%, but there will always be some spare. Not only does this allow for the possibility that there is a worse worst-case which you have not yet seen, but it also makes life easier for future development when new code needs to be added which uses more stack.

Answer 4

Stackoverflow is one of the many reasons for undefined behavior of a program. In this case you can get an expected result or segmentation fault or your hard disk could be erased, etc. Do not expect any defined behaviour because it's undefined behaviour.