What method call in HttpsURLConnection actually sends the HTTP request?

Question

EDIT: Now that I've figured out some behavior, I'm wondering, why does it not send the request as soon as I call flush()? or close()? Why does it wait until I call getResponseCode()?

---

Original question:

I am new to the HttpsURLConnection class in java, and looked on the doc and found it not very enlightening. I have inherited this code on a project I am working on (I did not write it myself) and wondered what it's doing, and how it can be improved. For example, I can't tell if "writeBytes" actually sends the data or "close." I've looked at many javadocs and found them to be ambiguous, and haven't found any good resources for this topic in books or blog posts online. Could someone please enlighten me, or point me to some good resources?

By the way, this code is for an android SDK library I am working on.

Note: I understand the theory of HTTP requests very well. I took a class on it. I know all about URL parameters (using the ?name=value) and cookies and RESTful services and TCP and IP... etc. Just struggling to find good docs so I know how to use the java libraries.

Edit: Changed this to the HttpClient code because it turns out I couldn't use Https and still see the request from the echo server. It has the same idea though.

import java.io.BufferedInputStream;
import java.io.DataOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.net.HttpURLConnection;
import java.net.MalformedURLException;
import java.net.URL;

public class HttpClient
{
    public static final String DEBUG_URL = "http://10.20.1.61:8001/api/ad/v5/";
    public static final String MAPPED_KEYWORDS = "get_mapped_keywords/";

    private static byte[] buffer = new byte[1024];

    static NetworkReturn sendHttpPost(String urlString, String postData)
    {
        URL url;
        HttpURLConnection con = null;

        try {
            url = new URL(urlString);
            con = (HttpURLConnection) url.openConnection();

            con.setRequestMethod("POST");
            con.setRequestProperty("Content-length", String.valueOf(postData.length()));
            con.setRequestProperty("Content-Type", "application/json");
            con.setDoOutput(true);

            DataOutputStream output = new DataOutputStream(con.getOutputStream());

            output.writeBytes(postData);
            output.close();

            StringBuilder result = new StringBuilder(), error = new StringBuilder();

            int responseCode = con.getResponseCode();
            if (responseCode >= 200 && responseCode < 300) 
            {
                if (con.getInputStream() != null) 
                {
                    InputStream in = new BufferedInputStream(con.getInputStream());
                    int length;
                    while ((length = in.read(buffer)) != -1) 
                    {
                        result.append(new String(buffer, 0, length));
                    }
                }
            }
            else 
            {
                if (con.getErrorStream() != null) 
                {
                    InputStream in = new BufferedInputStream(con.getErrorStream());
                    int length;
                    while ((length = in.read(buffer)) != -1) 
                    {
                        error.append(new String(buffer, 0, length));
                    }
                }
            }
            return new NetworkReturn(responseCode, result.toString(), error.toString());

        }
        catch (MalformedURLException e) {
            return new NetworkReturn(-1, "", "MalformedURLException: " + e.getLocalizedMessage());
        }
        catch (IOException e) {
            e.printStackTrace();
            return new NetworkReturn(-1, "", "IOEXCEPTION: " + e.getLocalizedMessage());
        }
        finally {
            if (con != null)
                con.disconnect();
        }
    }
}

Answer 1

The first method you call that gets input from the server will set the content-length header and write the output. That could be getResponseCode() or getInputStream(), or maybe getErrorStream().

If you're using fixed-length or chunked transfer mode, writing is direct.

Answer 2

I finally figured it out. The "getResponseCode()" method is the straw that breaks the camel's back (so to speak). Because when I commented out stuff, it was when I commented that part out it didn't "echo" my response on the echo server I was using (and printing the received requests to the terminal).

Note: I used a nearly-identical HttpClient class because otherwise the echo server would print out encrypted code like this:

����5D��s����l!���RMav*X?�+gJ.�+�/ � �����32E98�/A5� P localhost�

When I ran this code as a Java application, the server did not print anything to the terminal: (notice the commented out parts)

static NetworkReturn sendHttpPost(String urlString, String postData)
    {
        URL url;
        HttpURLConnection con = null;

        try {
            url = new URL(urlString);
            con = (HttpURLConnection) url.openConnection();

            con.setRequestMethod("POST");
            con.setRequestProperty("Content-length", String.valueOf(postData.length()));
            con.setRequestProperty("Content-Type", "application/json");
            con.setDoOutput(true);

            DataOutputStream output = new DataOutputStream(con.getOutputStream());
            output.writeBytes(postData);

            output.close();

            //StringBuilder result = new StringBuilder(), error = new StringBuilder();

            //int responseCode = con.getResponseCode();
            /*
            if (responseCode >= 200 && responseCode < 300) 
            {
                if (con.getInputStream() != null) 
                {
                    InputStream in = new BufferedInputStream(con.getInputStream());
                    int length;
                    while ((length = in.read(buffer)) != -1) 
                    {
                        result.append(new String(buffer, 0, length));
                    }
                }
            }
            else 
            {
                if (con.getErrorStream() != null) 
                {
                    InputStream in = new BufferedInputStream(con.getErrorStream());
                    int length;
                    while ((length = in.read(buffer)) != -1) 
                    {
                        error.append(new String(buffer, 0, length));
                    }
                }
            }
            return new NetworkReturn(responseCode, result.toString(), error.toString());
            */
            return new NetworkReturn();

        }
        catch (MalformedURLException e) {
            return new NetworkReturn(-1, "", "MalformedURLException: " + e.getLocalizedMessage());
        }
        catch (IOException e) {
            e.printStackTrace();
            return new NetworkReturn(-1, "", "IOEXCEPTION: " + e.getLocalizedMessage());
        }
        finally {
            if (con != null)
                con.disconnect();
        }
    }

However, this code DID make the server print to the terminal:

static NetworkReturn sendHttpPost(String urlString, String postData)
    {
        URL url;
        HttpURLConnection con = null;

        try {
            url = new URL(urlString);
            con = (HttpURLConnection) url.openConnection();

            con.setRequestMethod("POST");
            con.setRequestProperty("Content-length", String.valueOf(postData.length()));
            con.setRequestProperty("Content-Type", "application/json");
            con.setDoOutput(true);

            DataOutputStream output = new DataOutputStream(con.getOutputStream());
            output.writeBytes(postData);

            output.close();

            //StringBuilder result = new StringBuilder(), error = new StringBuilder();

            int responseCode = con.getResponseCode();
            /*
            if (responseCode >= 200 && responseCode < 300) 
            {
                if (con.getInputStream() != null) 
                {
                    InputStream in = new BufferedInputStream(con.getInputStream());
                    int length;
                    while ((length = in.read(buffer)) != -1) 
                    {
                        result.append(new String(buffer, 0, length));
                    }
                }
            }
            else 
            {
                if (con.getErrorStream() != null) 
                {
                    InputStream in = new BufferedInputStream(con.getErrorStream());
                    int length;
                    while ((length = in.read(buffer)) != -1) 
                    {
                        error.append(new String(buffer, 0, length));
                    }
                }
            }
            return new NetworkReturn(responseCode, result.toString(), error.toString());
            */
            return new NetworkReturn();

        }
        catch (MalformedURLException e) {
            return new NetworkReturn(-1, "", "MalformedURLException: " + e.getLocalizedMessage());
        }
        catch (IOException e) {
            e.printStackTrace();
            return new NetworkReturn(-1, "", "IOEXCEPTION: " + e.getLocalizedMessage());
        }
        finally {
            if (con != null)
                con.disconnect();
        }
    }

and the terminal output was this (an echo server, so this is what the request must have looked like):

POST / HTTP/1.1
Content-Type: application/json
User-Agent: Java/1.7.0_60
Host: localhost:3000
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive
Content-Length: 16

blah blah blah

This was the method call I did:

public class Main
{
    public static void main(String[] args)
    {
        NetworkReturn result = HttpClient.sendHttpPost("http://localhost:3000/", "blah blah blah\n\n");
    }
}

Edit: I tried Brett's suggestion by calling flush to the output stream (getResponseCode() is still commented out, so now I'm seeing if flush will make it work):

...

            DataOutputStream output = new DataOutputStream(con.getOutputStream());
            output.writeBytes(postData);
            output.flush(); //<------------here

            output.close();

            //StringBuilder result = new StringBuilder(), error = new StringBuilder();

            //int responseCode = con.getResponseCode();
            /*
            if (responseCode >= 200 && responseCode < 300) 
            {
                if
...

But my echo server didn't print anything. It makes sense, now that I think about it, because "output.close();" probably calls "flush()" already. If flush() worked, then my first code snippet (with getResponseCode() commented out) would have probably worked.

Edit: Tried to see if a 512 kilobyte request body would make the difference and cause the request to be sent, but it didn't work when getResponseCode() was commented out. I verified that it worked when I uncommented getResponseCode().

My new main method for testing a large "file", assuming a char is 1 byte so it should be over 512 kilobytes:

public class Main
{
    public static void main(String[] args)
    {
        StringBuilder largeString = new StringBuilder();
        for (int i = 0; i < 524288; i++)
        {
            largeString.append('a');
        }
        largeString.append("\n\n");
        NetworkReturn result = HttpClient.sendHttpPost("http://localhost:3000/", largeString.toString());

    }
}

Answer 3

For example, I can't tell if "writeBytes" actually sends the data or "close."

This is purposefully ambiguous to give implementations leeway.

The issue is whether or not fixed length or chunked requests will be used. It appears that fixed length requests are being used by default, though it does not appear that is required by the spec. The behavior can be controlled by methods on the HttpURLConnection. To specify use of chunked, you can call setChunkedStreamingMode. You can also call setFixedLengthStreamingMode to define the fixed length of the request. Even in this case, the implementation /could/ choose to start sending data early, as the total length is known and so can be set in the request header correctly.

Practically speaking, calling writeBytes will (generally) cause data to buffer until some threshold is reached. At that time the buffered data will be actually written and the process will repeat. When you call close, any buffered data will be written as well as the necessary bytes to signify the end of the http request.

Explicit calls to flush will also usually cause any buffered data to be written out.