You can't check for overflow of signed integers by performing the offending operation and seeing if the result wraps around.
First, the value 0x80000000
passed to the function is outside the range of a 32 bit int
. So it undergoes an implementation defined conversion. On most systems that use 2's compliment, this will result in the value with that representation which is -2147483648 which also happens to be the value of INT_MIN
.
Then you attempt to execute x - y
which results in signed integer overflow which triggers undefined behavior, giving you an unexpected result.
The proper way to handle this is to perform some algebra to ensure the overflow does not happen.
If x
and y
have the same sign then subtracting won't overflow.
If the signs differ and x
is positive, one might naively try this:
INT_MAX >= x - y
But this could overflow. Instead change it to the mathematically equivalent:
INT_MAX + y >= x
Because y is negative, INT_MAX + y
doesn't overflow.
A similar check can be done when x
is negative with INT_MIN
. The full check:
if (x>=0 && y>=0) {
return 1;
} else if (x<=0 && y<=0) {
return 1;
} else if (x>=0 && INT_MAX + y >= x) {
return 1;
} else if (x<0 && INT_MIN + y <= x) {
return 1;
} else {
return 0;
}