31

I'm trying to process a large graph using a recursive algorithm. Due to the deep recursion, I encountered the problem described at Python: Maximum recursion depth exceeded.

So, I tried increasing the limit on recursion depth, like so:

import sys
sys.setrecursionlimit(5000)

However, whatever value I use for the depth, I cannot get the result I want. Either I still get the exception, or else the program just halts with no output on the screen but: Process finished with exit code -1073741571.

How can I solve the problem?

See also: What is the hard recursion limit for Linux, Mac and Windows?

Karl Knechtel
  • 62,466
  • 11
  • 102
  • 153
Salvador Dali
  • 214,103
  • 147
  • 703
  • 753
  • 1
    By any chance, your program might need some optimization? – thefourtheye Dec 17 '13 at 08:13
  • CPython doesn't optimize tail recursion, so try to avoid using it to such an extent. – Blender Dec 17 '13 at 08:14
  • How many edges are in the graph? – Janne Karila Dec 17 '13 at 08:30
  • 1
    @thefourtheye There is nothing to optimize there. The only thing that is needed to be optimized is to change recursion to while loop :-) – Salvador Dali Dec 17 '13 at 09:22
  • @JanneKarila there are approximately 4500 edges in the graph – Salvador Dali Dec 17 '13 at 09:23
  • If you're exceeding the maximum recursion depth, it is very likely that your program is in an infinite recursion loop, in which case changing the recursion limit will be no help at all. You will need to figure out what should make your program complete its recursive step, and try to understand why that isn't happening. – Tim Pierce Dec 17 '13 at 20:56
  • @qwrrty It is not infinite. I tested it on many big graphs and it works correctly. Eventually it will finish. The problem is that in my case **setrecursionlimit** is not working. – Salvador Dali Dec 17 '13 at 21:57
  • http://docs.python.org/2/library/sys.html#sys.setrecursionlimit says: sys.**setrecursionlimit**(limit) - Set the maximum depth of the Python interpreter stack to limit. This limit prevents infinite recursion from causing an overflow of the C stack and crashing Python. The highest possible limit is platform-dependent. – CiaPan Mar 06 '14 at 23:08
  • @CiaPan this is a nice quotation, but what was your point? – Salvador Dali Mar 06 '14 at 23:12
  • 2
    Possibly you are trying to set the limit bigger than the environment allows. That would cause setrecursionlimit 'not work for you'. – CiaPan Mar 06 '14 at 23:22
  • @CiaPan thanks for clarification. The error with too high recursion limit looks different and I already checked that my recursion limit is much higher than I am using here. – Salvador Dali Mar 06 '14 at 23:26
  • Without the source code (and with no knowledge about python, actually) I can only guess... So here is my next guess: possibly you use mutual recursion, so the function A calls B, and B calls A for each step of the solution? That would double the actual number of invocations. – CiaPan Mar 06 '14 at 23:33
  • Yeah, I know. The problem is that there is no code anymore. After hopelessly trying to solve the problem I gave up and rewrote it without recursion. But there was not A calls B. It was just one function recursively calling itself. – Salvador Dali Mar 06 '14 at 23:45
  • Possibly it was not just a recursion depth but something like a recursion data volume? I'd try to write a test program to go a given depth down in a recursion as simple as possible anf test how far would it go. Then add some big set of variables to keep at each level and test again... But once you dropped the code there's no issue to test anymore. – CiaPan Mar 07 '14 at 09:32
  • The error code `-1073741571` in hex is `0xC00000FD`, which just means a `StackOverflow`. Since such errors cannot be caught, the process hard-exits. As others have already mentioned, the stack exhaustion comes before the runtime check thinks it is out of stack, leading to the hard error as opposed to the soft, catchable error. Btw, such exit codes can be caused in, or by many other applications as well. – Abel Jun 02 '20 at 18:27

2 Answers2

22

That is the signed integer representation of Microsoft's "stack overflow/stack exhaustion" error code 0xC00000FD.

You might try increasing your executable's stack size as described in the answer here: How to overcome Stack Size issue with Visual Studio (running C codes with big array)

Anytime you see strange, large negative exit codes in windows, convert them to hex and then look them up in the ntstatus error codes http://msdn.microsoft.com/en-us/library/cc704588.aspx

You may also use the MS Error Lookup Tool to find such codes locally, or even automatically. Other, similar tools exist both by Microsoft and third parties.

Abel
  • 56,041
  • 24
  • 146
  • 247
reteptilian
  • 903
  • 8
  • 12
  • 2
    This should be the accepted answer, it is correct. Thought it's important to mention that you check the `DWORD` (32-bit) value of the signed integer. If you accidentally convert it to `QWORD` (the default on Windows' Calc), it'll be left-padded with `FFFF FFFF` (or you can just remove those, the rest'll be the correct value). – Abel Jun 02 '20 at 18:24
16

You could use something like:

if __name__ == '__main__':
    sys.setrecursionlimit(100000)
    threading.stack_size(200000000)
    thread = threading.Thread(target=your_code)
    thread.start()

This solved both my recursion limitation and my heap size limitation.

Juan Pablo VEGA
  • 161
  • 1
  • 2
  • This removes the immediate symptom, which is sometimes fine; but it will not fix all recursion problems. In particular, if you have an endless recursion, you will run out of stack, no matter how large the stack is. – tripleee May 04 '21 at 07:09