Will a modern compiler such as GCC perform a tail case optimization in such cases?

Question

Assume I have a function like this :

int recursivefunction (int n) {
    if (n>10) return n; /* terminal case */
    elseif (n>5) return n*recursivefunction(n-1); /* non tail call*/
    else return recursivefunction(n-1); /* tail call

Will the third recursive call be optimized as a tail call ? If not why not ? We could replace this third return statement by an assembly code that clears the current function stack frame, except for local variables passed to the next call, and a goto to the adress of the function, this way no space on the stack is reserved.

Answer 1

Possibly, although it is not guaranteed.

The code is not quite an endless loop if n is less than or equsl to 5, but the eventual underflow when INT_MIN-1 is computed is Undefined Behaviour. If the compiler figures that out, it could do anything, including returning 42 or some other unspecifiable value without doing any call at all. This is also true of the second recursive call, since it must soon fall into the third one.

In short, there are no constraints on the compiler if the argument value is less than or equal to 10, because any of those cases will eventually perform a computation whose behaviour is undefined. So the compiler is entitled to assume that the programmer will ensure that the function will only be called with arguments greater than 10. If that is the case, the function is simply the identity function, so the compiler may remove all recursive calls, tail position or not.