Process virtual memory - space between stack and heap

Question

What happens when the unused memory space between stack and heap in a process's virtual memory is exhausted ?

Answer 1

In Windows there is a guard page between the stack and the heap and so the heap and the stack never touch. This is for security reasons (so that a stack exhaustion bug cannot lead to a more exploitable heap-overflow bug).

If you're asking what happens when the heap gets full, the simple answer is that your malloc (or HeapAlloc or whatever) calls will start returning NULL. If you're asking what happens when the stack gets full, well, you get a stack-exhaustion exception (sometimes wrongly called a stack-overflow). In either case this usually causes the program to free up some memory and continue or exit to the desktop.