0

I recently learned about Assembly x86 and how functions are implemented in it and how the stack program works.

However, I tried writing this program which calls a function f2 by changing the return address of the current called function f1, so that the instruction pointer starts f2 when finishing f1, therefore not returning directly to main.

It seems unstable and sometimes I get segmentation fault, while in another cases it works but does not return 0.

Why is that?

My guess is that the program stack is not given a contiguous space in memory at run time and so its behavior is not constant.

Sometimes it works if a change v[2] = (uintptr_t) f2; into v[another_index_greater_than_2] = (uintptr_t) f2;.

It is odd, since in theory v[1] should be the old base pointer pushed on the stack, while v[2] should be the return address of the function.

#include <iostream>

using namespace std;

int main();

void f2()
{
    int v[1];
    cout << "f2\n";
    v[2] = (uintptr_t) main;
}

void f1()
{
    int v[1];
    cout << "f1\n";
    v[2] = (uintptr_t) f2;
}

int main()
{
    f1();
    cout << "Back to main";
    return 0;
}

I expected to see the 3 strings printed in order (f1, f2, main) and the program to return 0, but the behavior of the program seems to be random.

Peter Cordes
  • 328,167
  • 45
  • 605
  • 847
Razvan48
  • 1
  • Perhaps there are more data on the stack than you expect? What is the compiler you're using? What is the target system? What is the ABI used? What does the stack-frame look like? – Some programmer dude Feb 23 '23 at 13:26
  • Also note that currently there are really no system with a 64-bit `int` type, while pointers on 64-bit systems are 64 bits. Storing 64 bits in a 32-bit type won't work that well. – Some programmer dude Feb 23 '23 at 13:27
  • I wrote the code in Windows CodeBlocks and compiled with GNU GCC – Razvan48 Feb 23 '23 at 13:27
  • As a test, compile the code `#include int main() { std::cout << sizeof(int*); }`. If the value is `8` then you are compiling for x64 and a pointers value wont fit in an int giving you signed integer overflow and undefined behavior. – NathanOliver Feb 23 '23 at 13:31
  • 3
    This is clearly Undefined Behavior, so any expectations will be incorrect. As one example, compiler can see out of bounds access and just ignore it completely. It *may* work for some specific version of specific compiler on specific ABI, but not portable in any way in general. – sklott Feb 23 '23 at 13:38
  • The most strange that it returns something at all. Any buffer v has no index 2 but at at most 0. – user10 Feb 23 '23 at 14:42
  • 1
    Since you're learning assembly, it would be a good exercise to read the assembly output from the compiler and try to figure out what is happening. Hacks like this usually fail because the programmer is assuming the compiler will compile the source code in the most straightforward and naive fashion (maybe because they are reading a tutorial based on compilers of 30 years ago), while in fact your modern compiler is doing optimizations that result in something else. Since the code has undefined behavior, the compiler can't really be "wrong" in doing so. – Nate Eldredge Feb 23 '23 at 15:19

1 Answers1

1

As @sklott said, the compiler may produce unexpected code by something like optimization. This answer assumes that it's expected output.

In the function prologue, rbp/ebp register is pushed first. Therefore, there is a pushed rbp/ebp register before the return address. If you compile for x64, the pushed rbp register is 8-bytes but int might be 4-bytes.

In this case, you're likely overwriting the higher 4-bytes on saved rbp. Maybe it'll cause f2 won't be run, and it'll return to main. The avoided rbp will be overwritten, it'll cause segmentation fault in some cases.

Assumed stack (without canary):
+--------+----------------+------+
|rsp+0x8 |                | v[0] |
+--------+----------------+------+
|rsp+0x10| saved rbp      | v[1] |
+--------+----------------+------+
|rsp+0x18| return address | v[2] |
+--------+----------------+------+

Actual Stack (without canary):
+--------+----------------+------+
|rsp+0xc |                | v[0] |
+--------+----------------+------+
|rsp+0x10| saved rbp      | v[1] |
|rsp+0x14|                | v[2] |
+--------+----------------+------+
|rsp+0x18| return address |      |
+--------+----------------+------+

To solve it, replace int v[1] with uintptr_t v[1];. However, (uintptr_t) main; will call f1(); again, so it'll be an infinite loop.

Note that the code won't work without adding -fno-stack-protector in GCC because canaries are enabled by default. However, since it seems to work in the question, I assume that it's added.

couyoh
  • 313
  • 3
  • 9
  • Are your diagrams supposed to be for x86-64? The register name is RBP; EBP is only its low half. Also, GCC or Clang will always keep RSP aligned by at least 8 at any point, normally to a 16-byte boundary if they reserve extra space for a local. So your final diagram that would leave the return address at `[rsp+0x14]` can't be right. Also, 8-byte RBP can't be split into two discontiguous chunks starting at 0x8 and 0x10. Maybe you meant 0xc and 0x10. – Peter Cordes Feb 23 '23 at 20:12
  • Anyway, yes GCC or clang might be reserving extra space on the stack, to keep it aligned by 16 (or even wasting an extra 16 bytes beyond that: [Why does GCC allocate more space than necessary on the stack, beyond what's needed for alignment?](https://stackoverflow.com/q/63009070)). But your actual 2nd diagram isn't plausible. https://godbolt.org/z/EfE71eGYT shows what GCC12 actually does; after `push rbp`, it does `sub rsp, 16` to keep RSP aligned, but `int v[]` is right below the saved RBP, so `v[2] = ...` stores to `[rbp+4]`. So yes, `uintptr_t v[1];` but then overwrite `v[1] = main` – Peter Cordes Feb 23 '23 at 20:18
  • With optimization enabled, `v[2] = ...` is optimized away entirely. It's just storing to a local variable which goes out of scope right after. GCC warns that it's past the end of the array, but that's undefined behaviour; there are no guaranteed semantics for this that GCC has to respect. Overwriting the return address isn't a thing in the C++ abstract machine, only an implementation detail of compiling without optimization for x86-64. – Peter Cordes Feb 23 '23 at 20:21
  • Your offsets still aren't right. If `v[0]` is at RSP+8 and `v[1]` is at RSP+0x10, `v` must have 8-byte elements. So `v[2]` would be at RSP+0x18. Or with 4-byte elements and 8-byte stack slots, v[0] and v[1] are at RSP+8, and v[2]+v[3] overlap the saved RBP. – Peter Cordes Feb 24 '23 at 01:22
  • Yup, that edit looks right. `int` definitely is 4 bytes on all mainstream compilers for x86. The code in the question was probably written for 32-bit mode, where `int` is the same size as `uintptr_t` and stack "slots". – Peter Cordes Feb 24 '23 at 03:21
  • @PeterCordes Thanks, I've changed the `rbp`'s offset from 0x8 to 0xc. – couyoh Feb 24 '23 at 03:22