37

Python's sys module provides a function setrecursionlimit that lets you change Python's maximum recursion limit. The docs say:

The highest possible limit is platform-dependent.

My question is: What is the highest possible limits for various platforms, under CPython? I would like to know the values for Linux, Mac and Windows.

UPDATE: Can we please avoid "You're doing it wrong" answers? I know that trying to do very deep recursion is usually a bad idea. I've considered the pros and cons in my specific situation and decided that I want to do it.

Georgy
  • 12,464
  • 7
  • 65
  • 73
Ram Rachum
  • 84,019
  • 84
  • 236
  • 374
  • 2
    Doesn't that also depend on how much memory you have, etc.?? – Justin May 26 '10 at 22:18
  • Likely equivalent to "what is the maximum possible setrlimit stack size on {os}". For Linux, I bet almost all of your userland RAM, but I don't know. Value can be manipulated with `resource.setrlimit`: http://stackoverflow.com/questions/5061582/setting-stacksize-in-a-python-script – Ciro Santilli OurBigBook.com Jan 28 '17 at 23:51

2 Answers2

36

On Windows (at least), sys.setrecursionlimit isn't the full story. The hard limit is on a per-thread basis and you need to call threading.stack_size and create a new thread once you reach a certain limit. (I think 1MB, but not sure) I've used this approach to increase it to a 64MB stack.

import sys
import threading

threading.stack_size(67108864) # 64MB stack
sys.setrecursionlimit(2 ** 20) # something real big
                               # you actually hit the 64MB limit first
                               # going by other answers, could just use 2**32-1

# only new threads get the redefined stack size
thread = threading.Thread(target=main)
thread.start()

I haven't tried to see what limits there might be on threading.stack_size, but feel free to try... that's where you need to look.

In summary, sys.setrecursionlimit is just a limit enforced by the interpreter itself. threading.stack_size lets you manipulate the actual limit imposed by the OS. If you hit the latter limit first, Python will just crash completely.

FogleBird
  • 74,300
  • 25
  • 125
  • 131
  • What should I put in place of 'target=main' if I don't have a 'main' object? – mmj Apr 18 '12 at 00:15
  • You put any function name there. It doesn't have to be called main. – FogleBird Apr 18 '12 at 01:08
  • Any value I put as stack size is not able to solve my recursion limit problem. I know that the real recursion depth I need is under 100k and I managed to run it under Linux, but not under Windows. – mmj Apr 18 '12 at 02:01
  • Works perfectly for me on Windows 10, Creators Update version. I can increase stack size this way up to 2^28-1 bytes (or 256 MB); not sure what the default is, but certainly much less than this. Why not more than 256MB? Perhaps some Windows limitation? – max Jun 10 '17 at 00:13
  • On macOS 10.14.5 there is the same problem and only this solution works! Thank you for saving my whole damn day! – Mr. Sun Lin May 31 '19 at 09:50
3

You shouldn't overuse recursive calls in CPython. It has not tail optimization, the function calls use a lot of memory and processing time. Those limits might not apply to other implementations, it's not in the blueprints.

In CPython, recursion is fine for traversing data structures (where a limit of 1000 should be enough for everybody) but not for algorithms. If I were to implement, say, graph related algorithms and hit the recursion limit, I would either implement my own stack and use iterations, or look for libraries implemented in C/C++/whatever before raising the limit by hand.

Marco Mariani
  • 13,556
  • 6
  • 39
  • 55
  • 11
    Thanks for the info, but it's more of an insightful comment than an answer. (And as an answer, it's of the "You're doing it wrong" type.) – Ram Rachum May 26 '10 at 22:33
  • Thanks @Xavier. fact is, in this profession, pretty much everything I learned from others was a shade of "you're doing it wrong". – Marco Mariani May 26 '10 at 23:01
  • I completely agree with you. You don't need recursion. You don't need stack limit. If you're causing Stack Overflow, you're doing it wrong. `=P` – Xavier Ho May 26 '10 at 23:17