7

We are using a python process to manage long running python subprocesses. Subprocesses occasionally need to be killed. The kill command does not completely kill the process, only makes it defunct.

Running the following script demonstrates this behaviour.

import subprocess
p = subprocess.Popen(['sleep', '400'], stdout=subprocess.PIPE, shell=False)

or

p = subprocess.Popen('sleep 400', stdout=subprocess.PIPE, shell=True)

Will create a subprocess.

p.terminate() 
p.kill()

does nothing to the process. Demonstrated by ps aux | grep sleep

$ ps aux| grep 'sleep'
User       8062  0.0  0.0   7292   764 pts/7    S    14:53   0:00 sleep 400

The process has not been killed/made defunct. Using the subprocess.call() function with 'kill' and pid as arguments will issue the kill command.

subprocess.call(['kill', str(p.pid)])

This will kill the process but it is now defunct.

$ ps aux | grep 'sleep'
User       8062  0.0  0.0      0     0 pts/7    Z+   14:51   0:00 [sleep] <defunct>

If the queue is running long enough will it eventually reach its maximum number of processes, or will it eventually reap the defunct processes and be fine?

If the answer is the former, how can I handle defunct processes in python without killing the parent process?

Is there a better way of killing processes?

user7495615
  • 73
  • 1
  • 1
  • 5
  • 1
    `kill` only sends a "signal" to the process and asks it to terminate it work. If you however `kill` it with `-9` it is killed by the operating system. But this is considered a bad way to end a process. – Willem Van Onsem Jan 31 '17 at 15:36
  • `kill -9` does the same thing as the `subprocess.kill()` function. Whether I try to terminate it cleanly with `p.terminate()` or kill it with `p.kill()` it still ends up as a zombie process. – user7495615 Jan 31 '17 at 15:53
  • I use `Popen.(['ffmpeg' etc...])` to save UDP data from localhost into an `mp3` file. When I end execution through the IDE everything works fine, but when I try to end it through the code itself with `Popen.terminate()` and `Popen.wait()` the file size is `zero`. – bomben Jul 02 '20 at 08:33

2 Answers2

22

There are 2 main issues here:

First issue: If you're using shell=True, so you're killing the shell running the process, not the process itself. With its parent killed, the child process goes defunct / isn't killed immediately.

In your case, you're using sleep which is not built-in, so you could drop shell=True, and Popen would yield the actual process id: p.terminate() would work.

You can (and you should) avoid shell=True most of the time, even if it requires extra python coding effort (piping 2 commands together, redirecting input/output, all those cases can be nicely handled by one or several Popen without shell=True.

And (second issue) if the process is still defunct when terminating after that fix, you could call p.wait() (from this question). Seems that calling terminate isn't enough. The Popen object needs to be garbage collected.

Community
  • 1
  • 1
Jean-François Fabre
  • 137,073
  • 23
  • 153
  • 219
  • 2
    Thanks, `p.wait()` after the `p.terminate()` call fixed it! – user7495615 Jan 31 '17 at 16:22
  • just for my information, can you try with `shell=True` to see if it still works? or if you get a defunct? – Jean-François Fabre Jan 31 '17 at 16:23
  • With shell=True the `ps aux | grep sleep` behaves as follows: `User 9689 0.0 0.0 4508 796 pts/7 S+ 16:28 0:00 /bin/sh -c sleep 100 User 9690 0.0 0.0 7292 648 pts/7 S+ 16:28 0:00 sleep 100` then issue the `p.kill()` or `p.terminate()` command. `$ ps aux| grep 'sleep' User 9690 0.0 0.0 7292 648 pts/7 S+ 16:28 0:00 sleep 100` – user7495615 Jan 31 '17 at 16:29
  • No I don't think that will work. There is still a `sleep` process running after issuing the `p.terminate()` command, the kill command has killed the shell command (i think!) – user7495615 Jan 31 '17 at 16:35
  • Is there something else going on? I use `Popen.(['ffmpeg' etc...])` to save UDP data from localhost into an `mp3` file. When I end execution through the IDE everything works fine, but when I try to end it through the code itself with `Popen.terminate()` and `Popen.wait()` the file size is `zero`. – bomben Jul 02 '20 at 08:28
  • it seems that it should be another question. – Jean-François Fabre Jul 02 '20 at 08:34
8

you should call p.wait() after terminating child process - for clearing process table. That should remove zombie processes (in defunct state)

VenkatC
  • 602
  • 3
  • 5