20

From the documentation:

sys.getrecursionlimit()

Return the current value of the recursion limit, the maximum depth of the Python interpreter stack. This limit prevents infinite recursion from causing an overflow of the C stack and crashing Python. It can be set by setrecursionlimit().

I am currently hitting the recursion limit when pickling an object. The object I am pickling only has a few levels of nesting, so I am a bit puzzled by what is happening.

I have been able to circumvent the issue with the following hack:

try:
    return pickle.dumps(x)
except:
    try:
        recursionlimit = getrecursionlimit()
        setrecursionlimit(2*recursionlimit)
        dumped = pickle.dumps(x)
        setrecursionlimit(recursionlimit)
        return dumped
    except:
        raise

Testing the above snippet on different contexts sometimes leads to success on the first try, and sometimes it leads to success on the second try. So far I have not been able to make it raise the exception.

To further debug my issue it would be helpful to have a way to obtain the current depth of the stack. That would allow me to verify if the entering stack depth is determining whether the snippet above will succeed on the first try or on the second.

Does the standard library provide a function to get the depth of the stack, or if not, how can I obtain it?

def get_stack_depth():
    # what goes here?
Darkonaut
  • 20,186
  • 7
  • 54
  • 65
Pedro M Duarte
  • 26,823
  • 7
  • 44
  • 43
  • 5
    This sounds like an XY problem. Instead of figuring out how to bypass the recursion limit you should try to figure out why you are hitting it. – kylieCatt Dec 06 '15 at 08:02
  • 3
    @IanAuld Indeed. That is exactly what I am trying to do, I am trying to see if the issue depends on the stack depth at the point where I make the offending call. To do that I need to figure out how to obtain the current depth of the stack. – Pedro M Duarte Dec 06 '15 at 08:04
  • What is the value of `x`? – kylieCatt Dec 06 '15 at 08:30

2 Answers2

34

If speed is an issue, it's way faster to bypass inspect module.

testing depth: 50 (CPython 3.7.3)
stacksize4b()         | depth: 50   |    2.0 µs
stacksize4b(200)      | depth: 50   |    2.2 µs
stacksize3a()         | depth: 50   |    2.4 µs
stacksize2a()         | depth: 50   |    2.9 µs
stackdepth2()         | depth: 50   |    3.0 µs
stackdepth1()         | depth: 50   |    3.0 µs
stackdepth3()         | depth: 50   |    3.4 µs
stacksize1()          | depth: 50   |    7.4 µs  # deprecated
len(inspect.stack())  | depth: 50   |    1.9 ms

I shortened the name of my functions to stacksize() and for easier differentiation, I'm referring to @lunixbochs' functions as stackdepth().


Basic Algorithms:

That's probably the best compromise between code brevity, readability and speed for small stack sizes. For under ~10 frames, only stackdepth1() is slightly faster due to lower overhead.

from itertools import count

def stack_size2a(size=2):
    """Get stack size for caller's frame.
    """
    frame = sys._getframe(size)

    for size in count(size):
        frame = frame.f_back
        if not frame:
            return size

For achieving better timings for larger stack sizes, some more refined algorithms are possible. stacksize3a() is combining chained attribute lookup with a close range finish from stackdepth1() for a much more favorable slope in timings, starting to pay off for roughly > 70 frames in my benchmarks.

from itertools import count

def stack_size3a(size=2):
    """Get stack size for caller's frame.
    """
    frame = sys._getframe(size)
    try:
        for size in count(size, 8):
            frame = frame.f_back.f_back.f_back.f_back.\
                f_back.f_back.f_back.f_back
    except AttributeError:
        while frame:
            frame = frame.f_back
            size += 1
        return size - 1

benchmark100

Advanced Algorithms:

As @lunixbochs has brought up in an answer, sys._getframe() is basically stackdepth1() in C-code. While simpler algorithms always start their depth-search in Python from an existing frame at the top of stack, checking the stack downward for further existing frames, stacksize4b() allows starting the search from any level by its stack_hint-parameter and can search the stack down- or upward if needed.

Under the hood, calling sys._getframe() always means walking the stack from the top frame downward to a specified depth. Because the performance difference between Python and C is so huge, it can still pay off to call sys._getframe() multiple times if necessary to find a frame closer to the deepest one, before applying a basic close-range frame-by-frame search in Python with frame.f_back.

from itertools import count

def stack_size4b(size_hint=8):
    """Get stack size for caller's frame.
    """
    get_frame = sys._getframe
    frame = None
    try:
        while True:
            frame = get_frame(size_hint)
            size_hint *= 2
    except ValueError:
        if frame:
            size_hint //= 2
        else:
            while not frame:
                size_hint = max(2, size_hint // 2)
                try:
                    frame = get_frame(size_hint)
                except ValueError:
                    continue

    for size in count(size_hint):
        frame = frame.f_back
        if not frame:
            return size

The usage-idea of stacksize4b() is to place the size-hint at the lower bound of your expected stack depth for a jump start, while still being able to cope with every drastic and short-lived change in stack-depth.

The benchmark shows stacksize4b() with default size_hint=8 and adjusted size_hint=200. For the benchmark all stack depths in the range 3-3000 have been tested to show the characteristic saw pattern in timings for stacksize4b().

benchmark300 benchmark3000

Darkonaut
  • 20,186
  • 7
  • 54
  • 65
  • @lunixbochs The usage-idea of the algorithm is to place the size-hint at the lower bound of your expected stack depth for a jump start, while still being able to cope with every drastic and short-lived change in stack-depth. The _lower_ bound and not the middle because triggered exceptions are relatively expensive, so the reverse-search of the algorithm should be limited. – Darkonaut Dec 30 '19 at 21:36
  • @lunixbochs It makes most sense if you have a bigger application which basically doesn't drop below a certain stack depth and you have a good reason to look into speed over code brevity in the first place, like you're very frequently trying to log callers (Python's dynamic nature makes this quite difficult for things like e.g. staticmethods), stack depth itself, or something like that. – Darkonaut Dec 30 '19 at 21:37
  • @lunixbochs Yeah, sounds like a reason. It depends on the expected stack size then. If you don't come over ~20 frames on average you should be fine with sticking with your answer for speed. – Darkonaut Dec 30 '19 at 21:58
  • 2
    Wow, this is an amazing answer. Thank you for diving deep into detail and showing all the pretty graphs. – theY4Kman Jan 20 '20 at 23:46
27

You can see the whole call stack from inspect.stack(), so currently taken depth would be len(inspect.stack(0)).

On the other hand, I guess you got the complete stack printed out when "maximum recursion depth exceeded" exception was raised. That stack trace should show you exactly what went wrong.

lunixbochs
  • 21,757
  • 2
  • 39
  • 47
zvone
  • 18,045
  • 3
  • 49
  • 77