Segmentation faults are instances of undefined behavior (UB). Read more about C, in particular the C11 standard n3337 and the Modern C book.
Regarding UB, read What every C programmer should know about undefined behavior
What is probably happening is that your buffer overflow or stack overflow happens on existing pages of your call stack. Read about page faults and virtual memory. Read a good textbook on operating systems.
You cannot be guaranteed that segmentation faults always happen or can be detected early. Because of ASLR, compiler optimizations, and Rice's theorem. Even the same executable could segfault on one run, and not on the next one.
Consider using valgrind or some address sanitizer.
Replace try[10]
with try[123456]
(or abc[0]
with abc[-456789]
....) and you might observe a different behavior of your process. Look also into the generated assembler code (e.g. obtained with gcc -Wall -Wextra -O2 -fverbose-asm -S
if using a recent and genuine GCC compiler; recent Clang accepts similar options, so you might try compiling with clang -Wall -Wextra -O2 -fverbose-asm -S
the same C source files).
If you have access to it, study the source code of the crt0 of your system.
Consider using static source code analysis tools such as Frama-C, or the Clang static analyzer. You could be interested in reading this draft report explaining some surprizing GCC optimizations.
You could install some Linux distribution on your laptop and take advantage of the fact that most software inside it are open source so study their source code. See also linuxfromscratch.org
Some source code of MacOSX is also open source (but not the GUI).