GCC's stack protection feature with optimization levels

Question

I am doing some experiments with GCC's stack protection feature to understand it better. Basically I referred to this post on stackoverflow.

The following is my code.

test.c

#include <stdio.h>

void write_at_index(char *arr, unsigned int idx, char val)
{
    arr[idx] = val;
    printf("\n%s %d arr[%u]=%d\n", __func__, __LINE__,
        idx, arr[idx]);
}

void test_stack_overflow()
{
    char a[16] = {0}; //Array of 16 bytes.

    write_at_index(a, 30/*idx*/, 10/*val*/); //Ask the other function to access 30th index.

    printf("\n%s %d Exiting a[0] %d\n", __func__, __LINE__, a[0]);
}

int main()
{
    test_stack_overflow();
    return 0;
}

The following is my makefile.

Makefile

CC=gcc

BIN=./test.out

SRCS=./test.c

all: $(BIN)

OBJ = ${SRCS:.c=.o}

CFLAGS=-O0 -fstack-protector -fstack-protector-all

$(OBJ): %.o: %.c
    $(CC) $(CFLAGS) $(INCLUDES) -c $*.c -o $*.o

$(BIN): $(OBJ)
    $(CC) -o $@ $<
    rm -rf ./*.o

clean:
    rm -rf ./*.out
    rm -rf ./*.o

I am using gcc (Ubuntu 7.3.0-27ubuntu1~18.04) 7.3.0

When I build and run test.out, I get " stack smashing detected " crash as expected.

However, if I change optimization level to O3, (CFLAGS=-O3 -fstack-protector -fstack-protector-all) and build and execute test.out, I am not observing the crash.

So my question is, does the compiler remove the "-fstack-protector" option when optimization is enabled? Or am I missing some other setting here?

The behaviour is *undefined*. Since none of the array writing is actually needed then the array can be optimized out and no stack smashing needs to happen. — Antti Haapala -- Слава Україні, Mar 20 '19 at 13:49
The optimization is good enough to remove the assembly code that caused stack smashing. — KamilCuk, Mar 20 '19 at 13:49
Prefix the functions with `static` and use Godbolt to explore — Antti Haapala -- Слава Україні, Mar 20 '19 at 13:50
@AnttiHaapala: Just for understanding - I am printing array[index] in my write_at_index(). So array writing is required right? — MayurK, Mar 20 '19 at 13:52
It is not required. The optimization can remove all the code and leave only calls to `printf`, as only this is a side effect. — KamilCuk, Mar 20 '19 at 13:56
Your code can be replaced by `puts("\nwrite_at_index 6 arr[30]=10\n\ntest_stack_overflow 16 Exiting a[0] 0")` and no one would notice any difference. — Antti Haapala -- Слава Україні, Mar 20 '19 at 13:58

score 1 · Answer 1 · answered Mar 20 '19 at 13:53

At higher optimization levels, write_at_index is inlined into test_stack_overflow, and GCC detects that the entire array a is not used in any meaningful way, and eliminates it. As a result, the store into the array and the associated buffer overflow are both gone. You would have to add a compiler barrier to prevent that from happening.

Furthermore, single-byte overflows only trigger stack protector crashes if they manage to hit the canary (or the return address). So they are not a good way to exercise stack overflow detection code.

GCC's stack protection feature with optimization levels

1 Answers1