Child process (by Popen) sometimes do not want to die if script finished

Question

I noticed that sometimes python does not kill child processes after Popen and finishing script.

The code:

proc = subprocess.Popen([blablabla_dir + 'blablabla'],
                        stdin=None, stderr=subprocess.STDOUT, stdout=subprocess.PIPE,
                        cwd = None,
                        env = {'LD_LIBRARY_PATH': blablabla_dir + 'lib'})
raise Exception()

Result:

...
    raise Exception()
Exception

But sometimes I have:

$ ps aux | grep blablabla
symonen+  2413  0.0  0.1 158572 11988 ?        Sl   19:04   0:00 /home/qqq/dev/eee/build/debug/blablabla
symonen+  2591  0.0  0.0  18636   944 pts/12   S+   19:04   0:00 grep --color=auto blablabla

So we have: if python "have time" to close process (after throw exception) then process finished. If not then not finished

So question: is there exist way to make force kill all child processes after finishing script (e.g. some option for script running or some else way). I'm making some tests for my applications and do not want to have garbage in system after finishing of my test's script.

I mean some ways only by Pyhon. So like "run script from bush script and then kill all garbage from bush script" - unsuitable.

Send to all created processes signal (15 or 9) from me is not good way. More precisely it is an option, but it does not solve the problem completely. Of course I closes my applications by signals. But: if we had unhandled exception before sending of kill then after script finishing we will have garbage again.

score 0 · Answer 1 · answered Dec 09 '15 at 19:47

I have a design idea which may help you solve your problem here. We will use one spawner python script to run spawn python scripts. Each spawn script runs a process and returns the process id (pid) to the spawner. The spawner is blocked while the spawn script is running, and once the spawn script either completes or fails, the spawner creates a call to kill the pid it received.

Some example code:

Spawner

import shlex
import subprocess
import re

if __name__ == "__main__":

    # Use shlex to make sure your arguments are parsed correctly
    # Here we have the command to run a spawn script
    args = shlex.split("python spawn.py")

    # Run the spawn script
    proc = subprocess.Popen(args,
                            stdout=subprocess.PIPE)

    # Waits for the spawn script to close and receives the output
    spawn_stdout, spawn_err = proc.communicate()

    # Go through the output to find the id of the process spawn started
    match = re.findall(r'(?<=PROCESS_ID)[\d]+?(?=END)', spawn_stdout)

    if len(match) > 0:
        kill = subprocess.Popen("kill {}".format(match[0]),
                                stdout=subprocess.PIPE,
                                shell=True)

Spawn

import subprocess

# We start the process
proc = subprocess.Popen([blablabla_dir + 'blablabla'],
                    stderr=None,
                    stdout=None)

# After it starts, print a unique string containing the process id
print "PROCESS_ID{}END".format(proc.pid)

This is a limited design and I'm sure you'll have to fiddle with it, I just wanted to get the idea out there. Hope it helps!

score 0 · Answer 2 · answered Dec 09 '15 at 21:17

0

Don't expect the child process to be killed when your program finishes. Neither subprocess nor "python" does that when the interpreter closes.

The proper way to do it is to catch the exception and kill your child processes, do it in a finally block, etc..

If you're concerned about any other processes spawned by your script you may want to kill the whole process group.

On Linux you can also build a really convoluted solution using ctypes and prctl(2) to make sure your children kill themselves when their parent dies. (Why is *nix terminology always so creepy? ;-))

answered Dec 09 '15 at 21:17

Karol Nowak

662
3
8

Good idea (prctl + PR_SET_PDEATHSIG). Big thanks for you. – Alexander Symonenko Dec 10 '15 at 09:19
Now I found correct topic about this: http://stackoverflow.com/questions/1884941/killing-the-child-processes-with-the-parent-process Thank you. – Alexander Symonenko Dec 10 '15 at 09:30
Is there exist some back-side effects if I will set PR_SET_PDEATHSIG to SIGTERM in my process? Usually it will work as daemon, also can be called from bash (with exports libs) in not daemon mode. – Alexander Symonenko Dec 10 '15 at 09:37
In you process, no, because you don't set it in your process. :) You have to do it after fork() and before exec(). This setting is not inherited by children of the child process. The only way to kill all these processes spawned by the shell script is to use process groups. – Karol Nowak Dec 10 '15 at 19:38

Child process (by Popen) sometimes do not want to die if script finished

2 Answers2