Why does increasing the recursion depth result in stack overflow error?

Question

I define a infinite recursive function as:

>>>def f():
>>>   f()
>>>

Then I called the function and this happend:

>>> f()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 2, in f
File "<stdin>", line 2, in f
File "<stdin>", line 2, in f
[Previous line repeated 996 more times]
RecursionError: maximum recursion depth exceeded
>>>

Next I do this:

>>>import sys
>>>sys.getrecursionlimit()
1000
>>>sys.setrecursionlimit(2147483647) #as 2147483647 is the highest number I can set for recursion in Python 3.8.5

Then I again call the function, but...

>>> f()
Traceback (most recent call last):
File "<stdin>", line 2, in f
File "<stdin>", line 2, in f
File "<stdin>", line 2, in f
[Previous line repeated 997 more times]
MemoryError: Stack overflow

I want to know, after changing the recursion limit to 2147483647 , why Python is still restricting the recursion to 1000?

Answer 1

The recursion limit was successfully updated, since the first errors message was:

RecursionError: maximum recursion depth exceeded

and then after increasing the recursion depth, the error message changed to:

MemoryError: Stack overflow

From the documentation on sys.setrecursionlimit():

Set the maximum depth of the Python interpreter stack to limit. This limit prevents infinite recursion from causing an overflow of the C stack and crashing Python.

Hence, by increasing the recursion limit, the program crashed the Python interpreter.

Since Python does not optimize tail recursion, unlimited recursion causes the stack to overflow (run out of memory).

In many real-world applications it is necessary to implement functions without recursion to avoid stack overflow memory errors.

See also: Setting stacksize in a python script