How do I increase the stack size in python

Question

I have a python program that uses a custom-built DLL. This DLL crashes due to a stack overflow. This overflow is not due to a recursive function gone bad, but to large allocations on the stack using alloca().

I want to increase stack size to get rid of this error. Is there any way to do this?

Answer 1

The python thread module allows you to specify a new stack size for new threads. Try setting that to a value you feel is large enough, and then doing the work of this DLL in a new thread.

Answer 2

As noted in some related questions like here, it's generally not a good idea to play with the stack size to extend the recursion depth, but here's code that shows how to grow the stack to that effect. With python 3.5, on a Windows 10 x64 system, it demonstrates a very deep recursion that's normally impossible (the normally allowed recursion limit in my situation appears to be 993). I don't know how big the stack actually has to be for this example, but, on my machine, with half of the size specified below, python crashes.

import sys
import threading

class SomeCallable:
    def __call__(self):
        try:
            self.recurse(99900)
        except RecursionError:
            print("Booh!")
        else:
            print("Hurray!")
    def recurse(self, n):
        if n > 0:
            self.recurse(n-1)

SomeCallable()() # recurse in current thread

# recurse in greedy thread
sys.setrecursionlimit(100000)
threading.stack_size(0x2000000)
t = threading.Thread(target=SomeCallable())
t.start()
t.join()

Answer 3

The functions in a dll can have no control over the stack size available when they are executed (unless you spawn new threads under the control of your library).

If the dll is custom, then can't you allocate on the heap rather than stack (or statically allocate, if appropriate), and stop the problem that way?

Answer 4

AFAIK a program can only change the stack size of new threads or processes (Windows' CreateThread function, for example). As Python (and the Win32 API for Python) does not expose such functionality, you should rather replace the stack allocation with heap memory. Or is there a specific reason for using the stack?? If you really have to use alloca you might want to create a separate thread for execution of DLL code (which is overkill I think).

EDIT: Correction - Python does allow for setting the stack size when creating new threads (see thread.stack_size)