python subprocesses (of process with subprocesses) on termination hang in OSX activity monitor

Question

I'm writing a test harness for a udp server I wrote. The idea is to use a multiprocessing.pool to do subprocess.call commands for several instances of the server (per this question). The udp server is another python program which consists of 3 processes (main, http interface and udp server). In my attempts to write a test script, when I kill the test script with CTRL+C, I always get 2-5 hanging python2.7 processes in my activity monitor.

My attempts:

1. vanilla subprocess
2. subprocess that handles bash not connected to terminal (see this question)
3. multiprocess.pool, also creates hanging process using simple test.py script (see below)

I've tried adding subprocess kills in the server code to no avail. When I run the server inside a terminal, it responds to CTRL+C correctly. I assume it's something with the child processes inside the server, because (1) and (2) have no problems with the test.py script.

---

1) just using subprocesses:

args0 = [sys.executable, server_path, '-p', '7000', '-t', '8000', '-u', '9000']
args1 = [sys.executable, server_path, '-p', '7001', '-t', '8001', '-u', '9001']

p0 = subprocess.Popen(args0)
p1 = subprocess.Popen(args1)

2) using subprocesses with stdout, pipe, os.kill:

p0 = subprocess.Popen(  args0, 
                        stdout=subprocess.PIPE, 
                        stderr=subprocess.STDOUT, 
                        close_fds=True)
print "running for 5s..."
time.sleep(5)
os.kill(p0.pid, signal.SIGTERM)

3a) test_harness.py (with pool)

import sys, os, time
import subprocess, multiprocessing

def work(cmd):
    return subprocess.call(cmd) # , shell=False

def main():
    server_path = os.getcwd() + '/' + 'test.py'

    args0 = [sys.executable, server_path, '-p', '7000']
    # args1 = [sys.executable, server_path, '-p', '7001']
    tasks = [args0]

    pool = multiprocessing.Pool(processes=1)
    pool.map_async(work, tasks)
    time.sleep(3)
    pool.terminate()

if __name__ == '__main__':
    main()

3b) test.py

def main():
    while True:
        a = 1

if __name__ == '__main__':
    main()

Answer 1

Got it working finally. This code spawns two multithreaded server instances, as subprocesses inside a test harness. You get get all console output from the subprocess, and when you CTRL+C the test harness from the terminal, all the subprocesses die too. subprocess.Popen didn't work for me ultimately.

def runInstance(args):
    # NOTE: We don't do the stdout or stderr args
    p = subprocess.call(args, 
                        # stdout=subprocess.PIPE, 
                        # stderr=subprocess.STDOUT, 
                        close_fds=True, 
                        shell=True
                        )

def main():
    print "Starting test harness..."
    server_path = os.getcwd() + '/' + 'server.py'
    args0 = [sys.executable, server_path, '-p', '7000', '-t', '8000', '-u', '9000']
    args1 = [sys.executable, server_path, '-p', '7001', '-t', '8001', '-u', '9001']

    # Start server instances
    # NOTE: It is target=runInstance, not target=runInstance()
    p0 = multiprocessing.Process(target=runInstance, args=(' '.join(args0),))
    p1 = multiprocessing.Process(target=runInstance, args=(' '.join(args1),))
    p0.start()
    p1.start()