Setting stacksize in a python script

Question

I am converting a csh script to a python script. The script calls a memory-intensive executable which requires a very large stack, so the csh script sets the stacksize to unlimited:

limit stacksize unlimited

When I try to reproduce this script in python, I execute them in a very naive manner, using os.system, e.g.:

os.system('some_executable')

But I do not know how to tell the OS to run these executables with unlimited stacksize. Is there a way to specify stacksize for calls within a python script? Is there some low-level system call that I should be using? And is there a module (similar to shutil) which controls this?

Attempting to set `rlimit_stack` after [Stack Clash](http://www.openwall.com/lists/oss-security/2017/06/19/1) remediations may result in failure or related problems. Also see Red Hat [Issue 1463241](https://bugzilla.redhat.com/show_bug.cgi?id=1463241) — jww, Jun 21 '17 at 16:21

Thomas Ahle · Accepted Answer · 2022-04-03T00:59:20.977

49

I have good experience with the following code. It doesn't require any special user permissions:

import resource, sys
resource.setrlimit(resource.RLIMIT_STACK, (2**29,-1))
sys.setrecursionlimit(10**6)

It does however not seem to work with pypy.

If resource.setrlimit doesn't work, you can also try using threads:

sys.setrecursionlimit(10**6)
import threading
threading.stack_size(2**26)
threading.Thread(target=main).start()

edited Apr 03 '22 at 00:59

answered Apr 27 '13 at 03:45

Thomas Ahle

30,774
21
92
114

7

`sys.setrecursionlimit(10**6)` will make the Python interpreter segfault if you overflow the stack. – Wilfred Hughes Jul 08 '14 at 15:11
7

Note that, as of this moment, the resource module is not available for Windows OS. – ofer.sheffer Aug 02 '14 at 18:21
@ofer.sheffer Still better than the accepted answer which uses `os.system`. – Thomas Ahle Sep 22 '14 at 17:07
1

@ThomasAhle I am `getting the error no module named resource` – garg10may Sep 25 '16 at 04:14
Try installing it. I have resource module – Szymon Roziewski Apr 14 '17 at 08:27
3

@ThomasAhle what is the `-1`? Did you mean `resource.RLIM_INFINITY`? – Boris Verkhovskiy Dec 28 '19 at 19:41

score 21 · Answer 2 · answered Feb 21 '11 at 02:07

21

You can just use the (u)limit command of your shell, if you want:

os.system('ulimit -s unlimited; some_executable')

Or (probably better) use resource.setrlimit:

resource.setrlimit(resource.RLIMIT_STACK, (resource.RLIM_INFINITY, resource.RLIM_INFINITY))

answered Feb 21 '11 at 02:07

Nicholas Riley

43,532
6
101
124

1

The first option doesn't work for me; I believe it is because os.system executes commands in the basic shell, sh, rather than bash or tcsh. But your second option is exactly what I needed, thank you. (I can't tell who suggested `setrlimit` first so I am awarding this one, since it includes the explicit code. I am grateful for everyone's answers, though). – marshall.ward Feb 21 '11 at 03:26
1

Yeah, I should have mentioned that - some shells use limit, some use ulimit, some allow both; the explicit invocation is a much better idea. – Nicholas Riley Feb 21 '11 at 03:36

score 13 · Answer 3 · answered Feb 21 '11 at 02:04

You're looking for the Python setrlimit interface, resource.RLIMIT_STACK.

Note that standard users cannot raise their hard limits, only root (well, a process with the CAP_SYS_RESOURCE capability (see capabilities(7)) processes can raise their limits; so you may need to use the PAM pam_limits(8) limits.conf(5) file to raise the hard limits for the users in question.

score 4 · Answer 4 · edited May 05 '20 at 15:05

4

You can alter the stack size of the current process via threading.stack_size(), but I don't know if that will be correctly inherited by subprocesses. That interface also requires a specific stack size - "unlimited" isn't an option.

edited May 05 '20 at 15:05

Boris Verkhovskiy

14,854
11
100
103

answered Feb 21 '11 at 02:10

ncoghlan

40,168
10
71
80

Setting stacksize in a python script

4 Answers4

Linked