Java socket performance bottleneck: where?

Question

I recently started the development of an application making intensive usage of networking. A first attempt was made using RMI and for a couple of reasons, we switched over to pure sockets. However, when testing sockets over a network, or even on localhost, we dropped to a rate of 25 requests/second. While using RMI it was two orders of magnitude higher.

With a little more testing, we obtained following (for localhost):

• Sending always the same object: 31628 requests/sec
• Sending always a new object: 25 requests/sec
• Only object creation rate: 3-4 millions per second (so that is not the bottleneck)

Here is the client code: (the server side just replies an "ACK")

public static void main(String[] args) throws IOException, ClassNotFoundException
{
    Socket kkSocket = null;
    ObjectOutputStream out = null;
    ObjectInputStream in = null;


    kkSocket = new Socket("barium", 4444);
    out = new ObjectOutputStream(kkSocket.getOutputStream());
    in = new ObjectInputStream(kkSocket.getInputStream());


    long throughput;
    long millis;

    TcpRequest hello = null;


    throughput = 0;
    millis = System.currentTimeMillis();
    while (System.currentTimeMillis() < millis + 1000)
    {
        hello = new TcpRequest();
        hello.service = "hello";
        hello.payload = Math.random();
        throughput++;
    }

    System.out.println("-------------------------------------------------------");
    System.out.println("|        Objects created: " + (throughput)  + " requests/sec.");
    System.out.println("-------------------------------------------------------");


    throughput = 0;
    millis = System.currentTimeMillis();
    while (System.currentTimeMillis() < millis + 1000)
    {
        out.writeObject(hello);
        Object res = in.readObject();
        throughput++;
    }
    System.out.println("-------------------------------------------------------");
    System.out.println("|        Same object throughput: " + (throughput)  + " requests/sec.");
    System.out.println("-------------------------------------------------------");


    throughput = 0;
    millis = System.currentTimeMillis();
    while (System.currentTimeMillis() < millis + 1000)
    {
        hello = new TcpRequest();
        out.writeObject(hello);
        Object res = in.readObject();
        throughput++;
    }
    System.out.println("-------------------------------------------------------");
    System.out.println("|        New objetcs throughput: " + (throughput)  + " requests/sec.");
    System.out.println("-------------------------------------------------------");


    out.close();
    in.close();

    kkSocket.close();
}

The TcpRequest class is just a dummy class without anything special.

So, if creating object is fast, if sending it over the network is fast ...why on earth is sending a new object over the network so slow?!?!

And if you keep a same object and modify its content before sending it, you will also have high transfer rate ...but fall in the nasty pitfall:

When working with object serialization it is important to keep in mind that the ObjectOutputStream maintains a hashtable mapping the objects written into the stream to a handle. When an object is written to the stream for the first time, its contents will be copied to the stream. Subsequent writes, however, result in a handle to the object being written to the stream.

...which happened to us and caused some hours of debugging before figuring it out.

So basically ...how do you achieve high throughput with sockets? (...I mean, with RMI being a wrapper around it we were already two orders of magnitude higher!)

SOLVED:

By replacing:

out = new ObjectOutputStream(kkSocket.getOutputStream());

With:

out = new ObjectOutputStream(new BufferedOutputStream(kkSocket.getOutputStream()))

The performances are normal again (nearly the same high throughput as with the same object case)

Answer 1

Found it:

Instead of:

out = new ObjectOutputStream(kkSocket.getOutputStream());

You should use:

out = new ObjectOutputStream(new BufferedOutputStream(kkSocket.getOutputStream()));

And

out.flush();

when sending a message.

...makes a huge difference ...though I don't know exactly why.

Answer 2

I would not trust the results of that benchmark. It does not ensure that the JVM is warmed up; i.e. that classes are loaded, initialized and compiled to native code before measuring execution times. There is a good chance that native code compilation kicks in part way through running the benchmark, and this inflates the time taken by one or more of the loops.

Answer 3

The problem that you're running into isn't low throughput with sockets; it's slow default Java serialization.

When you're sending the same object over and over, it's only really being serialized once, and then a reference to it is being sent each time; this explains why it goes so quickly.

When you're creating a new object every time, that new object has to be serialized using the relatively slow Java serialization mechanism, which is also probably much more complex than what you need.

What you can do to improve this is either implement custom serialization code for your class, or make your own object serialization protocol that is external to the class (and use DataOutput instead of ObjectOutputStream).

This article has a lot of good info, even if it is somewhat dated: http://java.sun.com/developer/technicalArticles/Programming/serialization/ See the last part on Performance Considerations

Answer 4

You should set the TCP_NODELAY option when working with small packets. Otherwise they will get delayed in an attempt to optimize network communication (a.k.a. Nagle's algorithm).

    socket.setTcpNoDelay(true);

The reason that using a buffer helps is because the delay hits small packets only. So it's not the buffering per se that helps, but the size of the packet.