46

I need to limit the amount of time and cpu taken by external command line apps I spawn from a python process using subprocess.call , mainly because sometimes the spawned process gets stuck and pins the cpu at 99%.

nice and ulimit seem like reasonable ways to do this, but I'm not sure how they'd interact with subprocess.

  • The limits look something like:
    • Kill the process if it's taking more than 60 seconds
    • Limit it to 20% of cpu
  • I want to apply the resource limiting to the subprocess, not to the python process that's spawning the subprocesses.

Is there a way to apply nice and ulimit to the subprocess.call spawned process? Are there better python-native alternatives?

This is on a linux (ubuntu) system.

Mike Graham
  • 73,987
  • 14
  • 101
  • 130
Parand
  • 102,950
  • 48
  • 151
  • 186

3 Answers3

119

Use the preexec_fn parameter to subprocess.Popen, and the resource module. Example:

parent.py:

#!/usr/bin/env python

import os
import sys
import resource
import subprocess

def setlimits():
    # Set maximum CPU time to 1 second in child process, after fork() but before exec()
    print "Setting resource limit in child (pid %d)" % os.getpid()
    resource.setrlimit(resource.RLIMIT_CPU, (1, 1))

print "CPU limit of parent (pid %d)" % os.getpid(), resource.getrlimit(resource.RLIMIT_CPU)
p = subprocess.Popen(["./child.py"], preexec_fn=setlimits)
print "CPU limit of parent (pid %d) after startup of child" % os.getpid(), resource.getrlimit(resource.RLIMIT_CPU)
p.wait()
print "CPU limit of parent (pid %d) after child finished executing" % os.getpid(), resource.getrlimit(resource.RLIMIT_CPU)

child.py:

#!/usr/bin/env python

import os
import sys
import resource

print "CPU limit of child (pid %d)" % os.getpid(), resource.getrlimit(resource.RLIMIT_CPU)

parent.py will fork into a new process. In the new process, it will call setlimits(), then exec child.py. This means the resource will be limited in the child process, but not in the parent.

Output when running program:

./parent.py
CPU limit of parent (pid 17404) (-1, -1)
Setting resource limit in child (pid 17405)
CPU limit of parent (pid 17404) after startup of child (-1, -1)
CPU limit of child (pid 17405) (1, 1)
CPU limit of parent (pid 17404) after child finished executing (-1, -1)

This is in many cases a better solution than trying to use ulimit, since it's not always a good idea to spawn subprocess via shell, especially since it often causes ugly parameter quoting trouble.

unutbu
  • 842,883
  • 184
  • 1,785
  • 1,677
Erik Forsberg
  • 4,819
  • 3
  • 27
  • 31
  • Thanks Erik. It looks like this sets the limits on the python process, not on the external process? – Parand Nov 06 '09 at 20:04
  • The Python process and all of its children. (That is, the way it's supposed to be. ;) ) From the man page: Limits on the consumption of system resources by the current process and each process it creates may be obtained with the getrlimit() call, and set with the setrlimit() call. – Andrew Dalke Nov 07 '09 at 01:32
  • 3
    Yes, the resource package sets the limit on the python process (via setrlimit) - but in my example, it sets the limit on the subprocess created by subproces.Popen, before calling exec() to run the child. So, in the example, the calling process' limits is not affected, only the limits of the child. – Erik Forsberg Nov 07 '09 at 11:01
  • do you guys know where I can download this `resource` package ? (for a machine not connected to the Internet) – Kevin Nov 08 '10 at 14:50
  • The resource package is included in Python, but it's only available on Unix platforms. See http://docs.python.org/library/resource.html – Erik Forsberg Nov 22 '10 at 21:24
  • Thanks for the solution, it makes sense, but in this case I only need to limit the child process - the parent process actually needs to remain unlimited. – Parand Jul 01 '11 at 00:14
  • 7
    This answer is 100% correct, and should have been chosen as the correct answer. – Michael van der Westhuizen Feb 27 '12 at 14:41
  • 4
    good answer on resource limits but it doesn't address the use of `nice` – RichVel Jun 13 '13 at 06:04
  • 3
    `os.nice` should be added to the answer. – W.Mann May 22 '17 at 06:53
  • To also set the nice value of the child, just add the line `os.nice(N)` (with the apropriate `N`) to the `setlimits()` function. – Kai Petzke Sep 06 '21 at 10:33
  • @Parand You and @Andrew miscommunicated. It does change only the children process limits/niceness. `Popen` forks current process, so before it calls `exec` you have two Python processes. The code in the answer will set rlimits and niceness for the children _and all of the children it creates_ - the parent will be left with its own values. This answer is correct. – Błażej Michalik Jun 10 '22 at 00:37
  • Another thing that should be noted, is that using `preexec_fn` makes `Popen` thread- ***un***safe. Do not try to do anything clever inside the function you are passing to it, otherwise you might deadlock the interpreter. – Błażej Michalik Jun 10 '22 at 00:40
  • See "Is `preexec_fn` ever safe in multi-threaded programs? Under what circumstances?": https://stackoverflow.com/questions/72686010/is-preexec-fn-ever-safe-in-multi-threaded-programs-under-what-circumstances – Błażej Michalik Jul 26 '22 at 08:37
11

You can set limits for subprocesses with the ulimit and nice shell commands like this:

import subprocess
subprocess.Popen('ulimit -t 60; nice -n 15 cpuhog', shell=True)

This runs cpuhog with a limit of 60 seconds of CPU time and a niceness adjustment of 15. Note that there is no simple way to set a 20% CPU throttle as such. The process will use 100% CPU unless another (less nice) process also needs the CPU.

Ville Laurikari
  • 28,380
  • 7
  • 60
  • 55
  • Thanks Ville, the cpu throttling you describe works great. Do you know if it's possible to do the same thing specifying the command with the bracket syntax instead of as a string? – Parand Nov 06 '09 at 19:39
  • As far as I can tell, you have to pass the entire shell command in one string for something like this to work. – Ville Laurikari Nov 06 '09 at 19:53
  • 9
    This is really not the solution that should be marked as accepted answer. In combination with user-supplied parameters, this can easily end up opening a security hole. – W.Mann May 22 '17 at 06:50
11

Erik made it easy for me, but he forgot the nice part which Rich Pointed out. I find the psutil package nice(pun intended) but unfortunately less portable. Here is my take at the question:

import os
import psutil
import resource
import subprocess

def preexec_fn():
    pid = os.getpid()
    ps = psutil.Process(pid)
    ps.set_nice(10)
    resource.setrlimit(resource.RLIMIT_CPU, (1, 1))

print "mother pid", os.getpid()
p = subprocess.Popen(["./cpuhog.sh"], preexec_fn=preexec_fn)
p.wait()
print "mother still alive with pid", os.getpid()

Ville used the shell=True to which I'm somehow allergic. Perhaps I'm just old and grumpy here, but I try to avoid it!

mogul
  • 4,441
  • 1
  • 18
  • 22