1

I have observed stack smashing and have read the posts about it,like this Stack smashing detected. and I understood that its like we are trying to override the boundaries and gcc have prevention mechanisms for that. But in my case when I change the row index to cross the bounds its printings 0s as values but when the column index is exceeding the bounds it's causing stack smashing.

Following is the program with necessary comments 

#include<stdio.h>  
void main()  
{  
    int i,j;  
    i=0;  
    j=0;  
    int d[2][2]={{0,0},{0,0}};  
    for(i=0;i<2;i++)   //**when I put  i<=2 there is no stack smashing**     
    {  
        for(j=0;j<2;j++)   //**When I put  j<=2 there comes the  stack smashing**    
        {  
            if(i==j)  
            {  
                d[i][j]=1;  
            }   
            else  
            {  
                d[i][j]=0;  
            }  
            printf("%d  ",d[i][j]);  
        }  
        printf("\n");  
    }  
}

matrix$ ./a.out Output with (i<=2) and (j<2)

1  0  
0  1  
0  0

Out put with (i<=2) and (j<=2) matrix$ ./a.out 

1  0  0  
0  1  0  
0  0  1

* stack smashing detected *: ./a.out terminated Aborted (core dumped)

is there any inherent limit on the number of items that can be exceeded to cause stack smashing based on architecture. Or is this just random? Any explanation would help as I couldn't figure

achoora
  • 1,270
  • 1
  • 16
  • 32
  • 5
    Because undefined behavior is undefined. You do a bad thing that can break things. Sometimes things don't break immediately. Maybe there's a safety margin, maybe there's some alignment going on. Maybe you just happen to be lucky and avoid killing the canary when overflowing in one way, but not the other (the thing that detects stack smashing is called a canary). Don't write outside of the array you allocated. – Art Nov 01 '17 at 12:24
  • If your array indexes are out of bounds, then anything can happen. It can even appear to work fine. – Jabberwocky Nov 01 '17 at 12:27
  • 1
    @AntonAngelov I have read the https://stackoverflow.com/questions/1345670/stack-smashing-detected ,I am curious about why its not happening with row index and happening only with the column index. There must be something under the hood – achoora Nov 01 '17 at 12:27
  • 1
    @achoora yes, there is something under the hood. It's called "undefined behaviour". If you really want to find out what's going on, execute your program step by step in disassembly mode, but I'd rather no waste my time with this. BTW on my VS2017 I get "stack smashing" in both cases !! – Jabberwocky Nov 01 '17 at 12:29
  • 1
    @achoora Under the hood, write down the array indexes you write to in both versions of the loop. You will notice that they are different. Two programs that do different things will have different behaviors. – Art Nov 01 '17 at 12:30
  • 1
    In one case you kill the canary, in the other case you're unlucky and happen to miss the canary. – Art Nov 01 '17 at 12:31
  • @MichaelWalz I tried with valgrind , it lead me to , Process terminating with default action of signal 6 (SIGABRT) at 0x4E6F428: raise (raise.c:54) by 0x4E71029: abort (abort.c:89) by 0x4EB17E9: __libc_message (libc_fatal.c:175) by 0x4F5311B: __fortify_fail (fortify_fail.c:37) by 0x4F530BF: __stack_chk_fail (stack_chk_fail.c:28) by 0x4006C4: main (in /home/achoora/Desktop/CodeJig/matrix/a.out) When the column index caused stack smashing, but the question about why row index is saved still remains!! – achoora Nov 01 '17 at 12:32
  • @Art , I agree that once I cross the bounds it can be undefined but why I am crossing the bounds in one case and escaping without scratch in another should be deterministic isnt it . I ran my program using gcc compiler on a linux pc , didnt toss a coin to leave it to chance :D – achoora Nov 01 '17 at 12:35
  • @anchoora In this case this behavior could be specific to the particular compiler, compiler version and libc implementation. – Anton Angelov Nov 01 '17 at 12:37
  • @achoora Undefined behaviour is undefined. Valgrind basically just told you that you did something wrong. Don't waste your time. Writing out of array bounds is undefined behaviour. Use [`assert`](http://www.cplusplus.com/reference/cassert/assert/) for early detection of out of bounds problems. – Jabberwocky Nov 01 '17 at 12:38
  • @achoora _escaping without scratch in another should be deterministic isnt it?_. Well it _may_ be deterministic on _your_ computer with _your_ operating system and _your_ compiler. – Jabberwocky Nov 01 '17 at 12:40
  • @achoora BTW, how do you know that you escaped without a scratch? Actually there _is_ a scratch, you just don't see it. – Jabberwocky Nov 01 '17 at 12:41
  • @MichaelWalz Thats right , might be in my selection of gcc compiler and the amd processor based on x86 architecture lies some thing which causes this index mis match thing. Any guide to debugging using disassembly would be helpful , since this can be my starting point to it .any references??. Today is a holiday , I have some time :D . – achoora Nov 01 '17 at 12:44
  • I'm voting to close this question as off-topic because it's asking for an explanation of a specific undefined behavior which is by definition not possible. – Jared Smith Nov 01 '17 at 15:08

2 Answers2

5

Most implementations of stack smashing detection is done by the prologue of the function writing a magic value (often called "canary", like a canary in a coal mine) on the stack. Then, the epilogue of the function checks if the magic value is still the same. Nothing in a correct program can overwrite the magic value therefore if the value has changed, the program must broken and is quickly killed.

The two different variants of your program overwrite different parts of the stack. One of them happens to overwrite the canary. The other one doesn't.

You were unlucky and the bug detection mechanism didn't detect the bug in one version of your program. No bug detection is perfect.

Art
  • 19,807
  • 1
  • 34
  • 60
4

Writing out of bounds of an array invokes undefined behavior, meaning that anything can happen.

The compiler is free to arrange the memory layout of your local variables in any way it pleases. It can allocate them from top to bottom, from bottom to top, with lots of space in between or whatever. It doesn't need to allocate them on the stack - the computer doesn't even need to have a stack, as far as C is concerned. The compiler could also allocate the variables in register, or not allocate them at all.

This is why various garage hacker tutorials speaking about "stack smashing" from a generic C programming point of view are confused. In order to actually exploit memory layout, you need to know exactly how the layout is done in the very specific case.

On hosted systems, the stack use will be specified by the system ABI. It is common that the system uses so-called stack frames, which are system-specific. Various protection mechanisms of the specific system may or may not be present.

Another aspect of undefined behavior (a C programming term for things beyond the scope of the language), is that the compiler is free to assume that the programmer never invokes it. So when you declare an array of size 2, the compiler is free to generate code which assumes that you never access items beyond that.

Meaning that even though you wrote code that access 1 item beyond the array, the compiler is not even obliged to listen to the programmer and generate the corresponding machine code.

Lundin
  • 195,001
  • 40
  • 254
  • 396