4

I have a small web application which opens a TCP socket connection, issues a command, reads the response and then closes the connection for every request to a particular REST endpoint.

I've started load testing the endpoint using Apache JMeter and am noticing that after running for some time, I start seeing errors like "Cannot assign requested address", the code opening this connection is:

def lookup(word: String): Option[String] = {
 try {
  val socket = new Socket(InetAddress.getByName("localhost"), 2222)
  val out = new PrintStream(socket.getOutputStream)
  val reader = new BufferedReader(new InputStreamReader(socket.getInputStream, "utf8"))
  out.println("lookup " + word)
  out.flush()

  var curr = reader.readLine()
  var response = ""
  while (!curr.contains("SUCC") && !curr.contains("FAIL")) {
    response += curr + "\n"
    curr = reader.readLine()
  }
  socket.close()
  curr match {
    case code if code.contains(SUCCESS_CODE) => {
      Some(response)
    }
    case _ => None
  }
 }
 catch {
   case e: Exception => println("Got an exception "+ e.getMessage); None
 }
}

When I run netstat I also see a lot of the following TIME_WAIT connection statues, which implies to me that I am running out of ports in the ephemeral space.

tcp6       0      0 localhost:54646         localhost:2222          TIME_WAIT  
tcp6       0      0 localhost:54638         localhost:2222          TIME_WAIT  
tcp6       0      0 localhost:54790         localhost:2222          TIME_WAIT  
tcp6       0      0 localhost:54882         localhost:2222          TIME_WAIT

I am wondering what the best solution is for this issue. My current thought is that creating a connection pool where connections to this service running on port 2222 can be reused by different HTTP requests, rather than creating new requests each time. Is this a sensible way of fixing the issue and making the application scale better? It seems like a lot of overhead to introduce and definitely makes my application more complicated.

Are there any other solutions for helping this application scale and overcome this port issue that I'm not seeing? My web application is running in an Ubuntu linux VM.

Alexis Wilke
  • 19,179
  • 10
  • 84
  • 156
jcm
  • 5,499
  • 11
  • 49
  • 78
  • 1
    Do you need SO_REUSE_ADDR? See the answer here: http://stackoverflow.com/questions/14388706/socket-options-so-reuseaddr-and-so-reuseport-how-do-they-differ-do-they-mean-t – The Archetypal Paul Sep 25 '14 at 07:20
  • 1
    Also http://stackoverflow.com/questions/16014627/tcp-connection-cant-be-established-when-there-is-a-tcp-connection-with-state-t?rq=1 and a few million :) other related questions - see the list on the RHS -> – The Archetypal Paul Sep 25 '14 at 07:22

3 Answers3

6

Yes, creating a connection pool is a good solution. However, an easier solution is to have the server close connections, rather than the client. In this case, the server's sockets, rather than the client's, would end up in TIME_WAIT state, so the client would not run out of ports. On the server side, connections in TIME_WAIT state would not make the server run out of ports because they all use the same local port.

To make sure that the connection is closed by the server, you need to read from the socket (on the client) until you reach end-of-file condition. At this moment, it is safe to close the socket on the client side because the server has closed it already. Of course, you need to make sure that the server would close the socket, rather than, say, waiting for a new request.

Alternatively, if you have root access, there are some sysctl options that you can tweak:

  • net.ipv4.ip_local_port_range – range of ephemeral ports. Increase it to make more ports available for outgoing connections.
  • net.ipv4.tcp_tw_recycle – enable faster recycling of connections in TIME_WAIT state.
  • net.ipv4.tcp_tw_reuse – enable reuse of connections in TIME_WAIT state. Not recommended.

See man pages ip(7) and tcp(7) for more information.

abacabadabacaba
  • 2,662
  • 1
  • 13
  • 18
  • Hi, I don't have control over the server, and there is no 'end-of-file' condition which would cause the server to close the connection on its side. – jcm Oct 01 '14 at 12:31
  • @cm22 You may try sending some invalid data to make the server close its connection. I also added some sysctl options to the answer, but you need root access to change them. – abacabadabacaba Oct 01 '14 at 12:55
  • the tuning guide @ https://gatling.io/docs/current/general/operations/ helps solve this issue and gives detail commands to execute as well. Adding this comment, as the link aligns with the above answer by @abacabadabacaba. thanks – Satya Mar 28 '18 at 22:44
0

Connection-pooling will solve this problem for you.

user207421
  • 305,947
  • 44
  • 307
  • 483
  • Hmm, it's not HTTP though - I need to connect to a service running on port 2222 that speaks a proprietary protocol, not HTTP :( I've cleared this misunderstanding up by removing the 200 and 404 statuses and setting them back to the proprietary protocol responses. – jcm Sep 25 '14 at 01:13
  • 1
    You also mentioned 'REST' in the question which people often associate with RESTful HTTP/Web Services. Of course, that does not have to be the case, as in your example. – monex0 Oct 04 '14 at 02:21
  • @jcm I didn't say anything about HTTP. I don't know what makes you think that HTTP is the only protocol in the world you can use connection-pooling with. – user207421 Apr 28 '16 at 20:22
0

I want to point out that the net.ipv4.tcp_tw_recycle has been completely removed from Linux 4.12 per commit 95a22caee396

Linux will randomize timestamp offsets for each connection, making this option completely broken, with or without NAT.

When the remote host is a NAT device, the condition on timestamps will forbid all the hosts except one behind the NAT device to connect during one minute because they do not share the same timestamp clock. In doubt, this is far better to disable this option since it leads to difficult to detect and difficult to diagnose problems.

Source

zangw
  • 43,869
  • 19
  • 177
  • 214