Socket attached to a child Python subprocess isn't properly freed

Question

I'm writing a library for testing of HTTP services. In my tests (I'm using Python 3.4) I use http.server as an example HTTP service. It's run in a separate process created with the same Python interpreter, somewhat like this:

subprocess.Popen([sys.executable, '-m', 'http.server', '9090'])

The problem

After the http.server process is killed, its port isn't usable (seems like it's still taken) from the parent Python process. Why is that?

More details

There is no problem with creating a new process that will use the same port as the last one, but I can't start listening on it from the parent process. The port also isn't shown as available by port_for.

If a non-Python process (i.e. Mountebank) is spawned in the same way, the port is properly freed.

Also, if I never call http.server over HTTP after it's started, then the port will still be available.

So what? Does Linux keep the resources (socket) taken by my child Python process attached to my parent one because it's created from the same executable? Or is it Python's doing?

EDIT 1

There's one thing missing from your test, that is actually my scenario: acquiring the socket from parent process. I've updated your test (below) and it shows that the socket still isn't free after the server stops.

import subprocess
import sys
import requests
import time
import signal
import os
import socket

for sig in [signal.SIGINT,signal.SIGTERM,signal.SIGKILL]:
    #sp = subprocess.Popen([sys.executable, '-m', 'http.server', '9090'])
    sp = subprocess.Popen([sys.executable, '-m', 'SimpleHTTPServer', '9090'])
    time.sleep(1)
    r = requests.get("http://localhost:9090/")
    print(r.content)
    os.kill(sp.pid,sig)
    time.sleep(1)

    # test if the socket is free
    try:
        s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        s.bind(('localhost', 9090))
        print('socket is FREE')
    except Exception:
        print('socket is NOT FREE')

I've checked for Python 2 and 3. The situation is the same whether I use os.kill or Popen (as expected). About my code, you can check for port availability before this line. I have to warn that my code is only for Python 3.

EDIT 2

Attempting to connect to the socket with SO_REUSEADDR like Foon said actually works. But it doesn't answer my original question.

I still don't know why a port taken by a Python subprocess will remain taken for the parent process unless you use REUSEADDR. I've also checked spawning simple HTTP server from a different interpreter than the one running master process (master - Python3, slave - Python2) and using a different program - a simple Bottle service.

When I spawn Mountebank (NodeJS) or NetCat subprocesses their ports are available after their cleanup even without using REUSEADDR.

After reading this post I've got the idea that it may have something to do with http.server running on 0.0.0.0 and Mountebank running only on localhost, but that had no impact.

So the question remains. Is there a bug in Python? Or some strange connection even between separate interpreters? Or did I pick non-Python processes that handle cleanup really well and I could observer the same behavior even in some non-Python programs?

Answer 1

So Linux will by default keep TCP sockets open for a while after you kill the process. Note that some processes may clean up better than others. You have a couple of options:

1. Quit the http.server process more cleanly

2. Set allow_reuse_address. I don't know how to do that a python one liner, but if you have something like the following, you shouldn't see your issue:

   def run(server_class=http.HTTPServer, handler_class=http.BaseHTTPRequestHandler):
   
     server_address = ('', 8000)
     httpd = server_class(server_address, handler_class)
     httpd.allow_reuse_address = True #killed processes won't linger tcp sockets
     httpd.serve_forever()
   

   run()

Edit: I can't reproduce your error. Here's what I ran to attempt to recreate it; could you provide a similar example to show what you're trying.

import subprocess
import sys
import requests
import time
import signal
import os

for sig in [signal.SIGINT,signal.SIGTERM,signal.SIGKILL]:
    #sp = subprocess.Popen([sys.executable, '-m', 'http.server', '9090'])
    sp = subprocess.Popen([sys.executable, '-m', 'SimpleHTTPServer', '9090'])
    time.sleep(1)
    r = requests.get("http://localhost:9090/")
    print(r.content)
    os.kill(sp.pid,sig)
    time.sleep(1)

(This goes through and does the subprocess call [nb that I don't have easy access to python3 at the moment, hence swapping out http.server for the python2 equivalent SimpleHTTPServer], waits a second to make sure that's ready, then uses requests to make an http request (since part of the issue tends to only exist if there's a TCP connection), then I kill it (first time using SIGINT, aka control-c, second time using SIGTERM (kill -15), third time using SIGKILL (kill -9)). The output I get is the expected http (listing of files in the directory I'm running in as HTML), followed by a bit of an error message from doing the equivalent of keyboard interrupt, followed by two more http listings (doing kill -15 or kill -9 doesn't give it a chance to print out a stacktrace)).

Edit 3:

I would try this code where you try to have the main part grab it (unless you also need to avoid having to do lower level socket stuff, in which case you may just be out of luck) (the key changes are that A) I added a call to set socketopt to reuseaddr and B) I called s.close() at the end of the loop

test if the socket is free

try:
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    s.bind(('localhost', 9090))
    print('socket is FREE')
    s.close()
except Exception as ex:
    print('socket is NOT FREE')
    print (ex)