HTTP post body lost spuriously (HttpURLConnection & Jetty)

Question

I have a home grown protocol which uses HttpURLConnection (from Java 1.6) & Jetty (6.1.26) to POST a block of xml as a request and receive a block of xml as a response. The amounts of xml are approx. 5KB.

When running both sender and receiver on Linux EC2 instances in different parts of the world I'm finding that in about 0.04% of my requests the Jetty handler sees the xml request (the post body) as an empty string. I've checked and the client outputs that it's consistently trying to send the correct (> 0 length) xml request string.

I have also reproduced this by looping my JUnit tests on my local (Win 8) box.

I assume the error must be something like:

Misuse of buffers
An HttpURLConnection bug
A network error
A Jetty bug
A random head slapping stupid thing I've done in the code

The relevant code is below:

CLIENT

        connection = (HttpURLConnection) (new URL (url)).openConnection();
        connection.setReadTimeout(readTimeoutMS);
        connection.setConnectTimeout(connectTimeoutMS);
        connection.setRequestMethod("POST");
        connection.setAllowUserInteraction(false);
        connection.setDoOutput(true);

        // Send request
        byte[] postBytes = requestXML.getBytes("UTF-8");
        connection.setRequestProperty("Content-length", "" + postBytes.length);
        OutputStream os = connection.getOutputStream();
        os.write(postBytes);
        os.flush();
        os.close();

        // Read response
        InputStream is = connection.getInputStream();
        StringWriter writer = new StringWriter();
        IOUtils.copy(is, writer, "UTF-8");
        is.close();
        connection.disconnect();
        return writer.toString();

SERVER (Jetty handler)

    public void handle(java.lang.String target, javax.servlet.http.HttpServletRequest request, javax.servlet.http.HttpServletResponse response, int dispatch) {
        InputStream is = request.getInputStream();
        StringWriter writer = new StringWriter();
        IOUtils.copy(is, writer, "UTF-8");
        is.close(); 
        String requestXML = writer.toString();
        // requestXML is 0 length string about 0.04% of time

Can anyone think of why I'd randomly get the request as an empty string?

Thanks!

EDIT

I introduced some more trace and getContentLength() returns -1 when the error occurs, but the client output still shows it's sending the right amount of bytes.

After some more test runs, getContentLength() is -1 in the fail case. Will edit to reflect this. — Jonathan, Nov 08 '12 at 20:29
At this point, I'd bust out Wireshark on the server. Let us know what you find. :) — user314104, Nov 08 '12 at 20:33
I think you're right although I don't have time right now. I'll do that at some point and report back. — Jonathan, Nov 09 '12 at 16:44

score 2 · Answer 1 · answered Nov 01 '12 at 21:20

2

I can't think of why you are getting a empty string. Code looks correct. If you update you code to check for empty string and if found report the content-length and transfer-encoding of the request, that would be helpful to identify the culprit. A wireshark trace of the network data would also be good.

But the bad new is that jetty-6 is really end of life, and we are unlikely to be updating it. If you are writing the code today, then you really should be using jetty-7 or 8. Perhaps even jetty-9 milestone release if you are brave. If you find such and error in jetty-9, I'd be all over it like a rash trying to fix it for you!

answered Nov 01 '12 at 21:20

gregw

2,354
20
21

Thanks Greg... that's a good point, I'll try dropping in the latest Jetty 8 jar and leave the test hammering it over night. – Jonathan Nov 01 '12 at 21:46
Unfortunately in my testing the problem still occurred spuriously with Jetty 8.1.7 – Jonathan Nov 02 '12 at 12:37
I started outputting the content length header as you suggested and it's coming through as -1. I guess that means I should be looking at the client side. – Jonathan Nov 02 '12 at 14:46
Are you trying to scale up a lot of these connections asynchronously? If you are then you likely want to avoid using that URLConnection and use a library that handles scaling by design, like jetty-client or netty. – jesse mcconnell Nov 02 '12 at 18:48
I will be scaling up later but this is reproducible with no concurrency; one request at a time. – Jonathan Nov 08 '12 at 20:27
At this point I think I'm just going to have to accept the rare errors and use retries. At some point in the future I'll look again. I'm awarding you the bounty for most detailed answer & because your suggestions were good debugging ideas. – Jonathan Nov 09 '12 at 16:46

Brian Henry · Answer 2 · 2012-11-08T17:50:36.277

1

Make sure you set connection.setRequestProperty("Content-Type", "application/xml"); It's possible POST data may be discarded without some Content-type. This was the case when I replicated your problem locally (against a Grails embedded Tomcat instance), and supplying this fixed it.

edited Nov 08 '12 at 17:50

answered Nov 07 '12 at 17:23

Brian Henry

3,161
1
16
17

1

also, you might consider something like apache httpclient (now @ http://hc.apache.org/). Might abstract away some of this lower-level bother. – Brian Henry Nov 07 '12 at 17:24
also, good resource on just this subject: http://stackoverflow.com/questions/2793150/how-to-use-java-net-urlconnection-to-fire-and-handle-http-requests – Brian Henry Nov 07 '12 at 17:25
I agree if the problem was consistent but I don't see how it could explain this happening for a small minority off calls. – Jonathan Nov 08 '12 at 19:08

HTTP post body lost spuriously (HttpURLConnection & Jetty)

2 Answers2