Python subprocess script keeps running after it is done

Question

In one of my Django views, I am calling a python script and getting its pid with:

from subprocess import Popen
p = Popen(['python', 'script.py'])
mypid = p.pid

When trying to find out if the process still is running from another page, I use the following function on mypid (thanks to this question):

def doesProcessExist(pid):
    if pid < 0:
        return False
    try:
        os.kill(pid, 0)
    except OSError, e:
        return e.errno == errno.EPERM
    else:
        return True

No matter how long I wait, the process still shows up as running. The only thing that stops it, is if I spawn a new python script process with Popen. Is there anyway I can fix this? I am not sure if this is caused by Django not closing python properly after the script is finished or something else. In Ubuntu's process status manager, the process shows up as [python] <defunct>.

--

The problem is true for all script.py I have tried. I am currently using one as simple as:

from time import sleep
sleep(5)

score 2 · Answer 1 · answered Nov 23 '13 at 01:32

Really, what you're doing is wrong. When you use a high-level wrapper like a subprocess.Popen, you need to manage the process through that object. Just having the PID elsewhere isn't enough to manage it.

If you insist on dealing in PIDs instead of Popen objects, then you should use the low-level APIs in os.

Fortunately, you're not doing anything complicated, like creating pipes to talk to the child process. So, you can just launch it with your favorite spawn variant, then wait for it with waitpid or one of its variants.

I'm assuming you're doing this all in a single-process web server. If you're using a forking web server, where the other page could be in a different process, even using PIDs won't work. The parent process has to reap the child, not some other arbitrary process. If you want to make that work, you'll have to make things more complicated, and you're really going to have to learn about the Unix process model before anyone can explain it to you.

Sam Hartman · Answer 2 · 2013-11-23T01:17:25.440

1

Processes in Unix stick around until the parent waits for them. calling wait on the object returned by thepopen will wait for the process to be done and will wait for it so it goes away. Until you do that it will exist as a zombie process See this message for info on getting the process to go away in the background while your web server runs without waiting for it in a foreground thread/view.

So, let's say that you do

p = subprocess.Popen(...)

At some point you need to call p.wait()

edited Nov 23 '13 at 01:17

answered Nov 23 '13 at 01:05

Sam Hartman

6,210
3
23
40

What you need to call is `wait`, not `close`. Also, you need to call it on the `Popen` process object, not on any pipe(s) you got (which is good, because in this case he doesn't _have_ any pipes.) – abarnert Nov 23 '13 at 01:10
@aba Thanks; updated.Was confusing C's pclose with subprocess.Popen – Sam Hartman Nov 23 '13 at 01:22
The thing is that I am waiting for the process in another page than where it was created, so I do not have the subprocess to run either wait() or communicate() on, but only the pid of the subprocess. – dwitvliet Nov 23 '13 at 01:22

score 1 · Accepted Answer · answered Nov 25 '13 at 02:50

What you see is a zombie process. It doesn't keep running. It can't. It is dead. The only thing that is left is some info that allows for related processes to retrieve its status.

To find out whether a subprocess is alive without blocking, call p.poll(). If it returns None then the process is still alive, otherwise you can safely forget about it (it is already reaped by .poll()).

subprocess module calls _cleanup() function that reaps zombie processes inside Popen() constructor. So normally your script won't create many zombie processes anyway.

To see a list of zombie processes:

import os
#NOTE: don't use Popen() here
print os.popen(r"ps aux | grep Z | grep -v grep").read(),

Python subprocess script keeps running after it is done

3 Answers3