GCC generates the wrong instructions when I write a function which returns a reference to a local variable. I know perfectly that you shouldn't do that.
Here is the simple Code:
#include <stdio.h>
#include <stdlib.h>
int *func()
{
int a = 100;
return &a;
}
int main()
{
printf("%p\n", func());
}
The output of the program is "(nil)".
I just compiled this with "gcc sample.c" and disassembled the executable with gdb:
Dump of assembler code for function func:
0x00000000004004e6 <+0>: push %rbp
0x00000000004004e7 <+1>: mov %rsp,%rbp
0x00000000004004ea <+4>: movl $0x64,-0x4(%rbp)
0x00000000004004f1 <+11>: mov $0x0,%eax
0x00000000004004f6 <+16>: pop %rbp
0x00000000004004f7 <+17>: retq
End of assembler dump.
Dump of assembler code for function main:
0x00000000004004f8 <+0>: push %rbp
0x00000000004004f9 <+1>: mov %rsp,%rbp
0x00000000004004fc <+4>: mov $0x0,%eax
0x0000000000400501 <+9>: callq 0x4004e6 <func>
0x0000000000400506 <+14>: mov %rax,%rsi
0x0000000000400509 <+17>: mov $0x4005a4,%edi
0x000000000040050e <+22>: mov $0x0,%eax
0x0000000000400513 <+27>: callq 0x4003c0 <printf@plt>
0x0000000000400518 <+32>: mov $0x0,%eax
0x000000000040051d <+37>: pop %rbp
0x000000000040051e <+38>: retq
End of assembler dump.
As you can see the return value is 0. It should be -0x4(%rbp). I've found nothing which this explains. My guess was that the GCC developers wanted that this code fails as fast as possible (null pointer dereferencing) but this couldn't be. A compiler has to generate the right instructions. I've tested this with GCC 5.3.0.