What is stack smashing and how do I fix it?

Question

The purpose of this program is to determine if a number between 1 and 1000 is prime by testing its divisibility with the first 11 prime integers. The program functions properly with most inputs. However, when I input an integer such as 468, stack smashing is detected. What is stack smashing and how do I resolve the issue?

I've tried researching stack smashing but I can't find specific examples that relate to my program. I am unaware of alternative methods I could try to amend the program as I am relatively new to programming with C.

char divStatement[] = " is divisible by ";

if ((userInput % 31) == 0) {
    div31 = true;
    strcat(divStatement, "31, ");
}

if (div2 || div3 || div5 || div7 || div11 || div13 || div17 || div19 || div23 || div29 || div31) {
        divStatement[strlen(divStatement) - 2] = '.';
        divStatement[strlen(divStatement) - 1] = '\n';
        printf("%d%s", userInput, divStatement);
        printf("%d is not prime.\n", userInput);
}
else {
        printf("%d is prime.\n", userInput);
}

The output actually works properly. At the end of the program, however, the terminal outputs:

***stack smashing detected ***: ./a.out terminated 
Aborted

Answer 1

char divStatement[] = " is divisible by ";

if ((userInput % 31) == 0) {
    div31 = true;
    strcat(divStatement, "31, ");
}

divStatement is a char array exactly large enough to hold " is divisible by " plus a \0 terminator. You can't append "31, " to it because it doesn't have any extra slack space.

An easy fix is to give the array an explicit length that's large enough to handle anything your program might append.

char divStatement[1000] = " is divisible by ";

Answer 2

Stack smashing error is returned because of the memory protection mechanisms used by the compiler. You could disable this feature but this would lead to vulnerabilities that are exploited by the so called stack buffer overflow attack (see this computerfile's video on youtube).

You're not declaring your character array with a specific size, but because you're initializing it with a string literal the size of the divStatement character array is set to 18 bytes (17 for the characters in " is divisible by " and 1 for the "\0", the so-called null character used for string termination). From the C99 standard 6.4.5/5 "String Literals - Semantics" (as quoted in an answer to this question):

The multibyte character sequence is then used to initialize an array of static storage duration and length just sufficient to contain the sequence.

Loosely speaking, the compiler knows there should be only 18 bytes in memory reserved for this specific character array and hence that writing to array shouldn't modify data past its 18th byte (for security reasons). When you're concatenating the strings, you're effectively trying to write more than 18 bytes which triggers a safety mechanism which in turn results in an error.

Since you're only testing the numbers between 1 and 1000, and only with the first 11 prime numbers, you know that the prime number will be at most 3 characters long. Together with the ", " this adds extra 5 characters, hence 18 + 5 = 23 characters altogether. Thus, your problem is easily solved by declaring the divStatement explicitly as a character array of size 23, that is, as divStatement[23] = " is divisible by ". As for getting a more general feedback on your code you should check out the code review community.

Answer 3

This is not the answer to your question, but a partial answer to the question you should have asked. I am to tired to come up with a proper explanation. But you need something like this somewhere in your code:

if ((userInput % n) == 0)
  {
    printf("%d is divisible by %d\n", userInput, n);
    is_prime == 0;
  }

Having an array of all the 11 first prime numbers is probably also a good idea:

const int primes[] = {2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31};

I am not going to spoil all 7 fun by telling you how to wrap this in a loop, and how to declare/initialize n and is_prime. Have fun with your homework. I hope the deadline isn't too soon :-)

Answer 4

The definition of Stack Smashing, as posted at: https://www.techopedia.com/definition/16157/stack-smashing

Definition - What does Stack Smashing mean?

*Stack smashing is a form of vulnerability where the stack of a computer application or OS is forced to overflow. This may lead to subverting the program/system and crashing it.

A stack, a first-in last-out circuit, is a form of buffer holding intermediate results of operations within it. To simplify, stack smashing putting more data into a stack than its holding capacity. Skilled hackers can deliberately introduce excessive data into the stack. The excessive data might be stored in other stack variables, including the function return address. When the function returns, it jumps to the malicious code on the stack, which might corrupt the entire system. The adjacent data on the stack is affected and forces the program to crash.*

regarding:

char divStatement[] = " is divisible by ";

and

strcat(divStatement, "31, ");

The call to strcat() is trying to append the string: "31, " to an array that is only large enough to hold the string: " is divisible by ". The result is the array is overflowed. Such overflow is undefined behavior. In this case, it corrupted the stack, probably right where some stack frame or other linkage is located