18

I'm writing a Python module that wraps out a certain web service API. It's all REST, so relatively straightforward to implement.

However, I found a problem when it comes to unit testing: as I don't run the services I made this module for, I don't want to hammer them, but at the same time, I need to retrieve data to run my tests. I looked at SimpleHTTPServer, which would be OK.

I solved part of the issue I had, but now, since I can't seem to terminate the thread, I get issues with "address already in use" when launching the test application more than once.

Here's some sample code

PORT = 8001

handler = SimpleHTTPServer.SimpleHTTPRequestHandler

httpd = SocketServer.TCPServer(("", PORT), handler)

httpd_thread = threading.Thread(target=httpd.serve_forever)
httpd_thread.daemon = True
httpd_thread.start()

api_data = urllib.urlopen("http://localhost:8001/post/index.json")
print "Data start:"
print json.load(api_data)

Where "index.json" is a mock JSON file I made which substitutes the real thing. How can I clean things gracefully after the program terminates?

ideasman42
  • 42,413
  • 44
  • 197
  • 320
Einar
  • 4,727
  • 7
  • 49
  • 64

3 Answers3

12

Try using a subclass of TCPServer with allow_reuse_address set True:

import socketserver

class TestServer(socketserver.TCPServer):
    allow_reuse_address = True

...
httpd = TestServer(("", PORT), handler)
ideasman42
  • 42,413
  • 44
  • 197
  • 320
Peter Hansen
  • 21,046
  • 5
  • 50
  • 72
  • brilliant! nice and simple – Claudiu Mar 05 '13 at 02:25
  • This didn't work for me, but I was able to solve this by setting up the SocketServer in a setUpClass method instead of the setUp method. The address in use error seems to happen the second time the setUp method is called (which happens when the second test is run). – mlissner Jun 14 '13 at 20:05
5

Old thread but the answers here didn't help me, I'm using HTTPServer, and shutting down after each unit test (by default HTTPServer has allow_reuse_address = 1 set). However I still got the address already in use problem after calling shutdown. I fixed using:

from BaseHTTPServer import HTTPServer

class MyHTTPServer(HTTPServer):
    def shutdown(self):
        self.socket.close()
        HTTPServer.shutdown(self)

Not sure why this doesn't happen by default? May be this isn't optimal?

Nick Holden
  • 3,639
  • 3
  • 22
  • 12
  • Whilst this worked for me in that it avoided the address re-use issue when I tried to open my URL in a test I ended up with: select error (9, 'Bad file descriptor') when shutting down the server. Still looking for a solution that doesn't produce this exception, although my test did appear to work. Python 2.7.10. – Keeely Jun 12 '18 at 21:17
4

We use a server built on wsgiref. http://docs.python.org/library/wsgiref.html

It's very easy to add features to this server as we add unit tests.

We start the server with subprocess. http://docs.python.org/library/subprocess.html?highlight=subprocess#module-subprocess

We do not use threads for this kind of testing. Why? (1) Our unit test server is rather complex and we'd like to keep it completely isolated from client applications. (2) Our client applications will be separate processes on separate hardware, we need to be sure that we have realistic performance expectations for that configuration. (3) It's simpler. (4) It's portable across all platforms. (5) It's trivial to change from stand-alone unit testing to integration testing with a production-like server that's already running.

We actually have a small WSGI application that makes the server shutdown in a reasonably controlled manner so that the logs are shutdown properly.

S.Lott
  • 384,516
  • 81
  • 508
  • 779