16

I have read that HttpURLConnection supports persistent connections, so that a connection can be reused for multiple requests. I tried it and the only way to send a second POST was by calling openConnection for a second time. Otherwise I got a IllegalStateException("Already connected"); I used the following:

try{
URL url = new URL("http://someconection.com");
}
catch(Exception e){}
HttpURLConnection con = (HttpURLConnection) url.openConnection();
//set output, input etc
//send POST
//Receive response
//Read whole response
//close input stream
con.disconnect();//have also tested commenting this out
con = (HttpURLConnection) url.openConnection();
//Send new POST

The second request is send over the same TCP connection (verified it with wireshark) but I can not understand why (although this is what I want) since I have called disconnect. I checked the source code for the HttpURLConnection and the implementation does keep a keepalive cache of connections to the same destinations. My problem is that I can not see how the connection is placed back in the cache after I have send the first request. The disconnect closes the connection and without the disconnect, still I can not see how the connection is placed back in the cache. I saw that the cache has a run method to go through over all idle connections (I am not sure how it is called), but I can not find how the connection is placed back in the cache. The only place that seems to happen is in the finished method of httpClient but this is not called for a POST with a response. Can anyone help me on this?

EDIT My interest is, what is the proper handling of an HttpUrlConnection object for tcp connection reuse. Should input/output stream be closed followed by a url.openConnection(); each time to send the new request (avoiding disconnect())? If yes, I can not see how the connection is being reused when I call url.openConnection() for the second time, since the connection has been removed from the cache for the first request and can not find how it is returned back. Is it possible that the connection is not returned back to the keepalive cache (bug?), but the OS has not released the tcp connection yet and on new connection, the OS returns the buffered connection (not yet released) or something similar? EDIT2 The only related i found was from JDK_KeepAlive

...when the application calls close() on the InputStream returned by URLConnection.getInputStream(), the JDK's HTTP protocol handler will try to clean up the connection and if successful, put the connection into a connection cache for reuse by future HTTP requests.

But I am not sure which handler is this. sun.net.www.protocol.http.Handler does not do any caching as I saw Thanks!

atiruz
  • 2,782
  • 27
  • 36
Cratylus
  • 52,998
  • 69
  • 209
  • 339
  • In one case : http://stackoverflow.com/questions/2457538/several-requests-from-one-httpurlconnection?rq=1 in my case I getting too many request to my php file without any get/post parameter , where as only 1 request with get parameter, .... I don't know why , if HttpURLConnection is meant to send one request at a time then, sending too many requests to server ... ? – Bhuro Jul 16 '16 at 11:04

6 Answers6

17

Should input/output stream be closed followed by a url.openConnection(); each time to send the new request (avoiding disconnect())?

Yes.

If yes, I can not see how the connection is being reused when I call url.openConnection() for the second time, since the connection has been removed from the cache for the first request and can not find how it is returned back.

You are confusing the HttpURLConnection with the underlying Socket and its underlying TCP connection. They aren't the same. The HttpURLConnection instances are GC'd, the underlying Socket is pooled, unless you call disconnect().

user207421
  • 305,947
  • 44
  • 307
  • 483
  • 1
    I started looking the src code once I saw(using wireshark) that if I called disconnect() the TCP connection was still reused. There is a concurrent hashmap for the caching of url connections, and I see that when a new HttpURlConnection is created, if there is a connection in the pool it uses it (in the HttpURLConnection constructor). But I can not see how the connection is placed back in the keepAlive cache, when I close the streams (and read whole response). I see that the connection is returned if the request is a HEAD (http.finished() is called) but not for a POST. – Cratylus Aug 12 '10 at 06:15
  • You are right the serverSockets are pooled in the concurrentHashMap but once a request is about to be done, the socket is removed from the ClientVector so that a new thread has to create a new connection. But this socket must return in the pool. This seems to happen only in putInKeepAliveCache();. But this is not called for a 200 OK with a response body. How is it possible to see the same behavior in the TCP connection using disconnect or not? – Cratylus Aug 12 '10 at 06:23
  • No, the *Sockets* are pooled. There are no ServerSockets in this context. And what you've found is that maybe not all of them are pooled. – user207421 Aug 13 '10 at 09:12
  • What do you mean by the HttpUrlConnection instances are GC'd ? – Pacerier Jan 17 '12 at 03:20
  • @Pacerier GC = Garbage Collection. – user207421 Feb 04 '12 at 09:03
  • On Android at least, the sockets are pooled even when disconnect() is called, by default. – James Wald May 21 '14 at 21:15
  • @JamesWald Android [docs](https://developer.android.com/reference/java/net/HttpURLConnection.html) says to disconnecting releases the resources held by a connection so they may be closed or reused. Are you sure it always gets polled and reused and not disconnected. Do you have any reference for this? Ideally I would expect it to retain the socket connection of there are other connections using it too. – Aniket Thakur Apr 12 '17 at 15:59
  • @AniketThakur Newer versions of Android use okhttp which exhibits similar behavior: "URLs that share the same address may also share the same underlying TCP socket connection. Sharing a connection has substantial performance benefits: lower latency, higher throughput (due to TCP slow start) and conserved battery. OkHttp uses a ConnectionPool that automatically reuses HTTP/1.x connections and multiplexes HTTP/2 and SPDY connections." https://github.com/square/okhttp/wiki/Connections – James Wald Apr 13 '17 at 00:52
  • @user207421, I have used HttpURLConnection and TCP socket is reused for next subsequent requests. But socket is closed if I send request after few mins (not sure for exact timing). Is there any callback for socket close event or is there any other way to know that request is not sent using same socket ? – Khushbu Shah Mar 17 '20 at 09:37
7

From the javadoc for HttpURLConnection (my emphasis):

Each HttpURLConnection instance is used to make a single request but the underlying network connection to the HTTP server may be transparently shared by other instances. Calling the close() methods on the InputStream or OutputStream of an HttpURLConnection after a request may free network resources associated with this instance but has no effect on any shared persistent connection. Calling the disconnect() method may close the underlying socket if a persistent connection is otherwise idle at that time.

Jim Garrison
  • 85,615
  • 20
  • 155
  • 190
  • I have read this. My question is how the connection returns in the buffer to be shared by other instances. I can not find the code that returns the connection to the cache – Cratylus Aug 11 '10 at 18:36
  • 1
    I'm not sure what your concern is. Are you seeing behavior that leads you to suspect a bug in the implementation? – Jim Garrison Aug 11 '10 at 20:29
  • I started looking the src code once I saw(using wireshark) that if I called disconnect() the TCP connection was still reused. In the src I could not see the connection placed back in the keepAlive cache once I closed the input stream of the first request. So there was no difference if I called disconnect() or not. This worried me that perhaps there is some kind of issue in the implementation unless I am missing something – Cratylus Aug 12 '10 at 06:18
4

I found that the connection is indeed cached when the InputStream is closed. Once the inputStream has been closed the underlying connection is buffered. The HttpURLConnection object is unusable for further requests though, since the object is considered still "connected", i.e. its boolean connected is set to true and is not cleared once the connection is placed back in the buffer. So each time a new HttpUrlConnection should be instantiated for a new POST, but the underlying TCP connection will be reused, if it has not timed out. So EJP answer's was the correct description. May be the behavior I saw, (reuse of the TCP connection) despite explicitly calling disconnect() was due to caching done by the OS? I do not know. I hope someone who knows can explain. Thanks.

Cratylus
  • 52,998
  • 69
  • 209
  • 339
  • 1
    No I don't think OS can do this. There is a way to force closure, by either forcing use of HTTP 1.0 (which does not support persistent connections), or using "Connection: close" header, if you really do NOT want any sharing of underlying TCP connetion. – StaxMan Oct 20 '10 at 16:00
  • @StaxMan:I know about the Connection header in HTTP.My confusion was due to the fact, that even though I explicitly closed the underlying socket, the connection was still reused. In my mind the behavior I was expecting was a 2(max) sec delay due to 3-way handshake and an HTTP connection over a new TCP connection. I did not see that and I thought that it was something related to OS caching – Cratylus Oct 21 '10 at 08:06
  • @Cratylus You didn't close the underlying socket. You can't. Only the underlying implementation can do that. Closing the input stream of the connection certainly didn't close that socket. – user207421 Oct 19 '12 at 20:44
  • 1
    I am seeing the exact same thing (reuse of TCP connection despite explicitly calling HttpURLConnection.disconnect()). Can anyone explain under what circumstances calling disconnect() would NOT cause the closure of the underlying TCP connection? – Tom McIntyre Jul 11 '13 at 08:58
  • @TomMcIntyre If the underlying connection had already been grabbed from the pool for use by another HttpURLconnection, it would be highly improper for disconnect() to close it. – user207421 Dec 22 '13 at 03:28
4

How do you "force use of HTTP1.0" using the HttpUrlConnection of JDK?

According to the section „Persistent Connections” of the Java 1.5 guide support for HTTP1.1 connections can be turned off or on using the java property http.keepAlive (default is true). Furthermore, the java property http.maxConnections indicates the maximum number of (concurrent) connections per destination to be kept alive at any given time.

Therefore, a "force use of HTTP1.0" could be applied for the whole application at once by setting the java property http.keepAlive to false.

Jakub M.
  • 32,471
  • 48
  • 110
  • 179
deadhorse
  • 41
  • 1
1

Hmmh. I may be missing something here (since this is an old question), but as far as I know, there are 2 well-known ways to force closing of the underlying TCP connection:

  • Force use of HTTP 1.0 (1.1 introduced persistent connections) -- this as indicated by the http request line
  • Send 'Connection' header with value 'close'; this will force closing as well.
StaxMan
  • 113,358
  • 34
  • 211
  • 239
  • How do you "force use of HTTP1.0" using the HttpUrlConnection of JDK? – Cratylus Oct 21 '10 at 08:03
  • Hmmh. Good question -- I assumed there was a way, but I could not find one with quick googling. So I would use the second solution; it's better either way, and for this purpose works as well. – StaxMan Oct 21 '10 at 20:09
0

Abandoning streams will cause idle TCP connections. The response stream should be read completely. Another thing I overlooked initially, and have seen overlooked in most answers on this topic is forgetting to deal with the error stream in case of exceptions. Code similar to this fixed one of my apps that wasn't releasing resources properly:

HttpURLConnection connection = (HttpURLConnection)new URL(uri).openConnection();
InputStream stream = null;
BufferedReader reader = null;
try {
        stream = connection.getInputStream();
        reader = new BufferedReader(new InputStreamReader(stream, Charset.forName("UTF-8")));

        // do work on part of the input stream

} catch (IOException e) {

    // read the error stream
    InputStream es = connection.getErrorStream();
    if (es != null) {
        BufferedReader esReader = null;
        esReader = new BufferedReader(new InputStreamReader(es, Charset.forName("UTF-8")));
        while (esReader.ready() && esReader.readLine() != null) {
        }
        if (esReader != null)
            esReader.close();
    }

    // do something with the IOException
} finally {

    // finish reading the input stream if it was not read completely in the try block, then close
    if (reader != null) {
        while (reader.readLine() != null) {
        }
        reader.close();
    }

    // Not sure if this is necessary, closing the buffered reader may close the input stream?
    if (stream != null) {
        stream.close();
    }

    // disconnect
    if (connection != null) {
        connection.disconnect();
    }
}

The buffered reader isn't strictly necessary, I chose it because my use case required reading one line at a time.

See also: http://docs.oracle.com/javase/1.5.0/docs/guide/net/http-keepalive.html

Crutis
  • 49
  • 7
  • You don't need to read the stream to EOS, or the error stream either. You just have to close. See your own citation. Closing the buffered reader will close the stream so you don't need that either. -1 – user207421 Sep 07 '13 at 20:52