How to clean up after subprocess.Popen?

Question

I have a long-running python script with a perl worker subprocess. Data is sent in and out of the child proc through its stdin and stdout. Periodically, the child must be restarted.

Unfortunately, after a while of running, it runs out of files ('too many open files'). lsof shows many remaining open pipes.

What's the proper way to clean up after a Popen'd process? Here's what I'm doing right now:

def start_helper(self):
    # spawn perl helper
    cwd = os.path.dirname(__file__)
    if not cwd:
        cwd = '.'

    self.subp = subprocess.Popen(['perl', 'theperlthing.pl'], shell=False, cwd=cwd,
                                 stdin=subprocess.PIPE, stdout=subprocess.PIPE,
                                 bufsize=1, env=perl_env)

def restart_helper(self):
    # clean up
    if self.subp.stdin:
        self.subp.stdin.close()
    if self.subp.stdout:
        self.subp.stdout.close()
    if self.subp.stderr:
        self.subp.stderr.close()

    # kill
    try:
        self.subp.kill()
    except OSError:
        # can't kill a dead proc
        pass
    self.subp.wait() # ?

    self.start_helper()

When you kill it, could it be raising an OSError for any reason other than already being dead? — Thomas K, Feb 17 '11 at 19:26
here's a somewhat unrelated bug in `subprocess` but it could shed some light on `_cleanup()` http://bugs.python.org/issue1731717 — jfs, Feb 17 '11 at 19:52
use `close_fds=True` (default on py3k) if you're not on Windows — jfs, Feb 17 '11 at 19:54
btw, on some systems there is a ridiculously small limit on number of open files. — jfs, Feb 17 '11 at 19:55
@J.F. Sebastian, I've added close_fds and will see, but it looks like that cleans up the fds from the master proc in the child, not vice-versa? — dkuebric, Feb 17 '11 at 20:01
Have you tried checking what's on the other end of those open pipes? Here's an unsophisticated hack which relies on NODE being the 8th field of lsof output: `lsof | grep " $$(lsof -p YOUR_PYTHON_PID | grep pipe | awk '{if (FNR > 1) {printf "%s", "\\|"} printf "%s", $8}')$ "` If the other ends are your perl processes, then you have a leak somehow... — rlibby, Feb 20 '11 at 06:32
I hate to say it, but I really want to know why the perl script has to periodically be killed and restarted in the first place! — Michael Scott Shappe, Feb 23 '11 at 21:53
So the python process is running out of open file descriptors? Are you running the child perl process multiple times in the parent? It's not clear why the parent process would be using additional file descriptors from this example. — stderr, Feb 24 '11 at 18:29

score 7 · Accepted Answer · answered Mar 29 '11 at 01:55

7

I think that's all you need:

def restart_helper(self):
    # kill the process if open
    try:
        self.subp.kill()
    except OSError:
        # can't kill a dead proc
        pass

    self.start_helper()
    # the wait comes after you opened the process
    # if you want to know how the process ended you can add
    # > if self.subp.wait() != 0:
    # usually a process that exits with 0 had no errors
    self.subp.wait()

As far as I know all file objects will be closed before the popen process gets killed.

answered Mar 29 '11 at 01:55

Thomas Stachl

418
2
8

The wait() is the key to not leaving subprocesses and open pipes behind. – dkagedal Mar 29 '11 at 22:22
I think this ended up being true. – dkuebric Aug 30 '11 at 22:55

score 1 · Answer 2 · answered Mar 01 '11 at 08:25

A quick experiment shows that x = open("/etc/motd"); x = 1 cleans up after itself and leaves no open file descriptor. If you drop the last reference to a subprocess.Popen the pipes seem to stick around. Is it possible you are re-invoking start_helper() (or even some other Popen) without explicitly closing and stopping the old one?

How to clean up after subprocess.Popen?

2 Answers2