2

I am currently learning reverse engineering using the x86 Instruction Set and had the general understanding that the first local variable of a function is always stored at the address [ebp-4]. However, the following C function doesn't seem to follow that principle and I'm struggling to find out why:

int check_authentication(char *password) {
    int auth_flag = 0;
    char buffer[16];
    strcpy(buffer, password);
    if(strcmp(buffer, "password") == 0)
        auth_flag = 1;
    return auth_flag; 
}

For the line int auth_flag = 0, I was expecting the corresponding x86 ASM translation to be mov DWORD PTR[ebp-4], 0. However, the first lines of the compiled ASM output of the above function look like below (compiled via gcc -m32 -O0 [...]):

check_authentication:
        push    ebp
        mov     ebp, esp
        sub     esp, 40
        mov     DWORD PTR [ebp-12], 0 ; <- Why ??

Why does it use the memory address [ebp-12] and not [ebp-4]?
I figured this is related to function calls within the function (i.e., if I remove the calls to strcpy and strcmp, then [ebp-4] is used as expected).
I cannot seem to figure out why this is happening in the first place though.

imperfect perfectionist
  • 71
  • 1
  • 7
  • 2
    "general understanding that the first local variable of a function is always stored at the address [ebp-4]" : There is no such general principle. The order in which variables were declared does not necessarily have any effect on where they are located on the stack (if at all). A compiler, especially an optimizing compiler, will rearrange them freely. – Nate Eldredge Apr 03 '22 at 20:43
  • 1
    Extra unused space is often added for [alignment](https://stackoverflow.com/questions/49391001/why-does-the-x86-64-amd64-system-v-abi-mandate-a-16-byte-stack-alignment). Additionally, `gcc -O0` doesn't optimize stack usage and will often choose an inefficient arrangement that wastes space unnecessarily. – Nate Eldredge Apr 03 '22 at 20:44
  • @NateEldredge thanks for your answer, but I still don't have a clear understanding. Removing `gcc -O0` doesn't change this behavior. Also, why does the removal of function calls within my function above (i.e., removing strcpy and strcmp), lead to the compiled x86 code using `[ebp-4]` again? – imperfect perfectionist Apr 03 '22 at 20:53
  • 1
    Function calls require stack alignment (see my link above), and so a function which calls other functions will have a different stack layout than a "leaf function" which makes no calls. – Nate Eldredge Apr 03 '22 at 22:11
  • 2
    As for `-O0`, it's the default, so `gcc` by itself would have the same effect. But try compiling with `-O` or `-O2` or `-O3` and you'll see a different layout. In particular `auth_flag` will be optimized into a register so it will not require stack space at all. – Nate Eldredge Apr 03 '22 at 22:12
  • 3
    It is not really worth thinking hard about exactly why the compiler lays out the stack in a particular way. There are certain constraints that must be met: each variable has a required alignment depending on its type, function arguments must be pushed in the correct order and following the ABI, alignment is required at function calls. But otherwise, the compiler's choice of stack layout is more or less arbitrary. You could read the compiler source code to learn its precise algorithm if you care, but it isn't really useful information for any practical purpose. – Nate Eldredge Apr 03 '22 at 22:15
  • 1
    Related: [How to remove "noise" from GCC/clang assembly output?](https://stackoverflow.com/q/38552116) - look at optimized output, and realize that the dumb anti-optimized output you're seeing is just what compilers happen to do sometimes. (Also [Why does clang produce inefficient asm with -O0 (for this simple floating point sum)?](https://stackoverflow.com/q/53366394) re: exactly what `-O0` does/doesn't imply. Nothing about layout details of the stack frame.) And of course -O1 or higher won't waste instructions setting up EBP as a frame pointer in the first place. – Peter Cordes Apr 04 '22 at 00:44
  • 2
    BTW, you've already found a counterexample to your title question, so you know the answer is "no". A better way to phrase it might be "why does GCC -O0 usually store the first local at [ebp-4], and why not always?" – Peter Cordes Apr 04 '22 at 00:46
  • 1
    [Is gcc reordering local variables at compilation time?](https://stackoverflow.com/q/38891137/995714), [What is the order of local variables on the stack?](https://stackoverflow.com/q/51867757/995714), [Order of local variable allocation on the stack](https://stackoverflow.com/q/1102049/995714), [How the local variable stored in stack](https://stackoverflow.com/q/50524734/995714), [Why does gcc reorder the local variable in function?](https://stackoverflow.com/q/36298567/995714) – phuclv Apr 04 '22 at 01:28
  • Thank you for all the clarifications, all of them helped me a lot, also the links / references to other StackOverflow questions. @PeterCordes I agree, I have reframed the question as suggested by you. – imperfect perfectionist Apr 04 '22 at 09:33

1 Answers1

2

C is not a high-order assembly. You cannot expect the compiler to compile C to assembly statement by statement.

Is the first local variable in x86 ASM always stored at location [ebp-4]?

No, that would tie the hands of the C compiler authors too much.

phuclv
  • 37,963
  • 15
  • 156
  • 475
Joshua
  • 40,822
  • 8
  • 72
  • 132