GCC optimizer generating error in nostdlib code

Question

I have the following code:

void cp(void *a, const void *b, int n) {
    for (int i = 0; i < n; ++i) {
        ((char *) a)[i] = ((const char *) b)[i];
    }
}

void _start(void) {
    char buf[20];

    const char m[] = "123456789012345";
    cp(buf, m, 15);

    register int rax __asm__ ("rax") = 60; // exit
    register int rdi __asm__ ("rdi") = 0; // status

    __asm__ volatile (
        "syscall" :: "r" (rax), "r" (rdi) : "cc", "rcx", "r11"
    );

    __builtin_unreachable();
}

If I compile it with gcc -nostdlib -O1 "./a.c" -o "./a", I get a functioning program, but if I compile it with -O2, I get a program that generates a segmentation fault.

This is the generated code with -O1:

0000000000001000 <cp>:
    1000:   b8 00 00 00 00          mov    $0x0,%eax
    1005:   0f b6 14 06             movzbl (%rsi,%rax,1),%edx
    1009:   88 14 07                mov    %dl,(%rdi,%rax,1)
    100c:   48 83 c0 01             add    $0x1,%rax
    1010:   48 83 f8 0f             cmp    $0xf,%rax
    1014:   75 ef                   jne    1005 <cp+0x5>
    1016:   c3                      retq   

0000000000001017 <_start>:
    1017:   48 83 ec 30             sub    $0x30,%rsp
    101b:   48 b8 31 32 33 34 35    movabs $0x3837363534333231,%rax
    1022:   36 37 38 
    1025:   48 ba 39 30 31 32 33    movabs $0x35343332313039,%rdx
    102c:   34 35 00 
    102f:   48 89 04 24             mov    %rax,(%rsp)
    1033:   48 89 54 24 08          mov    %rdx,0x8(%rsp)
    1038:   48 89 e6                mov    %rsp,%rsi
    103b:   48 8d 7c 24 10          lea    0x10(%rsp),%rdi
    1040:   ba 0f 00 00 00          mov    $0xf,%edx
    1045:   e8 b6 ff ff ff          callq  1000 <cp>
    104a:   b8 3c 00 00 00          mov    $0x3c,%eax
    104f:   bf 00 00 00 00          mov    $0x0,%edi
    1054:   0f 05                   syscall 

And this is the generated code with -O2:

0000000000001000 <cp>:
    1000:   31 c0                   xor    %eax,%eax
    1002:   66 0f 1f 44 00 00       nopw   0x0(%rax,%rax,1)
    1008:   0f b6 14 06             movzbl (%rsi,%rax,1),%edx
    100c:   88 14 07                mov    %dl,(%rdi,%rax,1)
    100f:   48 83 c0 01             add    $0x1,%rax
    1013:   48 83 f8 0f             cmp    $0xf,%rax
    1017:   75 ef                   jne    1008 <cp+0x8>
    1019:   c3                      retq   
    101a:   66 0f 1f 44 00 00       nopw   0x0(%rax,%rax,1)

0000000000001020 <_start>:
    1020:   48 8d 44 24 d8          lea    -0x28(%rsp),%rax
    1025:   48 8d 54 24 c9          lea    -0x37(%rsp),%rdx
    102a:   b9 31 00 00 00          mov    $0x31,%ecx
    102f:   66 0f 6f 05 c9 0f 00    movdqa 0xfc9(%rip),%xmm0        # 2000 <_start+0xfe0>
    1036:   00 
    1037:   48 8d 70 0f             lea    0xf(%rax),%rsi
    103b:   0f 29 44 24 c8          movaps %xmm0,-0x38(%rsp)
    1040:   eb 0d                   jmp    104f <_start+0x2f>
    1042:   66 0f 1f 44 00 00       nopw   0x0(%rax,%rax,1)
    1048:   0f b6 0a                movzbl (%rdx),%ecx
    104b:   48 83 c2 01             add    $0x1,%rdx
    104f:   88 08                   mov    %cl,(%rax)
    1051:   48 83 c0 01             add    $0x1,%rax
    1055:   48 39 f0                cmp    %rsi,%rax
    1058:   75 ee                   jne    1048 <_start+0x28>
    105a:   b8 3c 00 00 00          mov    $0x3c,%eax
    105f:   31 ff                   xor    %edi,%edi
    1061:   0f 05                   syscall 

The crash happens at 103b, instruction movaps %xmm0,-0x38(%rsp).

I noticed that if m contains less than 15 characters, then the generated code is different and the crash does not happen.

What am I doing wrong?

Answer 1

_start is not a function. It's not called by anything, and on entry the stack is 16-byte aligned, not (as the ABI requires) 8 bytes away from 16-byte alignment.

(The ABI requires 16-byte alignment before a call, and call pushes an 8-byte return address. So on function entry RSP-8 and RSP+8 are 16-byte aligned.)

---

At -O2 GCC uses alignment-required 16-byte instructions to implement the copy done by cp(), copying the "123456789012345" from static storage to the stack.

At -O1, GCC just uses two mov r64, imm64 instructions to get bytes into integer regs for 8-byte stores. These don't require alignment.

---

Workarounds

Just write a main in C like a normal person if you want everything to work.

Or if you're trying to microbenchmark something light-weight in asm, you can use gcc -nostdlib -O3 -mincoming-stack-boundary=3 (docs) to tell GCC that functions can't assume they're called with more than 8-byte alignment. Unlike -mpreferred-stack-boundary=3, this will still align by 16 before making further calls. So if you have other non-leaf functions, you might want to just use an attribute on your hacky C _start() instead of affecting the whole file.

---

A worse, more hacky way would be to try putting
asm("push %rax"); at the very top of _start to modify RSP by 8, where GCC hopefully runs it before doing anything else with the stack. GNU C Basic asm statements are implicitly volatile so you don't need asm volatile, although that wouldn't hurt.

You're 100% on your own and responsible for correctly tricking the compiler by using inline asm that works for whatever optimization level you're using.

---

Another safer way would be write your own light-weight _start that calls main:

// at global scope:
asm(
   ".globl _start \n"
   "_start:       \n"
   "    mov   (%rsp), %rdi  \n"     // argc
   "    lea   8(%rsp), %rsi  \n"    // argv
   "    lea   8(%rsi, %rdi, 8), %rdx \n"   // envp
   "    call  main \n"
          // NOT DONE: stdio cleanup or other atexit stuff
          // DO NOT USE WITH GLIBC; use libc's CRT code if you use libc
   "    mov   %eax, %edi \n"
   "    mov   $231, %eax \n"
   "    syscall"               // exit_group( main() )
);

int main(int argc, char**argv, char**envp) {
   ... your code here
   return 0;
}

If you didn't want main to return, you could just pop %rdi; mov %rsp, %rsi ; jmp main to give it argc and argv without a return address.

Then main can exit via inline asm, or by calling exit() or _exit() if you link libc. (But if you link libc, you should usually use its _start.)

See also: How Get arguments value using inline assembly in C without Glibc? for other hand-rolled _start versions; this is pretty much like @zwol's there.