0

I'm seeing some odd behaviour trying to warm up a pool of connections that uses java.net.Socket to connect.

Here is a little program that connects a lot of sockets:

public class SocketTest {
    public static void main(String[] args) throws Exception {
        for (int i = 0; i < 4000; i++) {
            long t = System.currentTimeMillis();
            new Socket("localhost", 3306);
            long total = System.currentTimeMillis() - t;
            if (total > 10)
                System.out.format("%5d %dms\n", i, total);
        }
    }
}

I tried this against the open ports on my machine.

MySQL:

  0 34ms
  468 1000ms
  676 997ms
  831 998ms
  970 997ms
  ...

ipp:

    0 41ms
  231 998ms
  232 998ms
  233 999ms
  234 999ms
  236 3002ms
  238 3002ms
  240 3002ms
  ...

eJabberd (xmpp server):

    0 45ms
   27 998ms
   42 999ms
   81 997ms
   99 1000ms
  120 997ms
  135 998ms
  147 997ms

MongoDB:

    0 73ms
 1314 999ms
 1791 998ms
 2098 999ms
 2466 1000ms
 2717 1000ms

nginx does not exhibit this behaviour (and connects very quickly).

In a multi-threaded test I always see only numbers that are very close to 1000ms, 3000ms, 7000ms or 15000ms.

Putting a Thread.sleep(50) between the socket connects delays the behaviour occurring for a while.

I originally observed this behaviour on Windows but the above was tested on XUbuntu 14.04.

I profiled the application and the time is spent in PlainSocketImpl.socketConnect() which delegates to the OS-native method on unix systems.

I also tried with both the Oracle JDK 8 and OpenJDK 7, and get the same results.

The consistency of the numbers makes me think it's some sort of throttling or scheduling behaviour at the operating system layer rather than something built into the server.

In smaller cases like connecting 100 sockets, the total time could be 10 seconds, but drops to 2 seconds with Thread.sleep(10) in between.

Can anyone shed some light?

EDIT: Tried it with Windows. Sometimes get similar numbers (3000) and sometimes get sporadic 500ms. Against MySQL on Windows the problem doesn't occur at all. Again, throwing in a quick Thread.sleep(20) yields far better performance, especially against remote machines.

I would guess that this is deliberate operating system throttling behaviour but would like to hear from somebody familiar with an OS stack.

bcoughlan
  • 25,987
  • 18
  • 90
  • 141
  • You could try *closing* them some time, for a start, instead of just leaking descriptors. – user207421 May 06 '15 at 00:48
  • I'm not familiar with Java; check what state your sockets are in after moving to the next iteration of the loop. It's also likely there is some sort of connection rate limiting, probably via `iptables` or `ufw`. I'd be more interested in how Windows behaves. – Collin Dauphinee May 06 '15 at 07:48
  • @CollinDauphinee Edited the post with some windows behaviour – bcoughlan May 06 '15 at 09:24
  • This sounds like IPv6 resolution is being done on localhost. Try connecting to 127.0.0.1 instead – Collin Dauphinee May 06 '15 at 16:45
  • @CollinDauphinee Just tried that, results are the same. I don't even need this any more, just dying to know now :) – bcoughlan May 07 '15 at 07:38
  • This makes me suspect it's Java-specific, or something in MySQL. I can't think of any type of time-out or rate limiting on Windows that would behave like this. I'll test with native sockets later. – Collin Dauphinee May 07 '15 at 18:12
  • 1
    This is certainly something specific to your machine, Java, or MySQL. I can connect to any local services at 127.0.0.1 on my Windows 7 box in <10ms, until I run out of ephemeral ports. – Collin Dauphinee May 08 '15 at 04:04

0 Answers0