I'm trying to use Java's HttpURLConnection to do a "conditional get", but I never get a 304 status code

Question

Here is my code:

    final HttpURLConnection conn = (HttpURLConnection) sourceURL.openConnection();
    if (cachedPage != null) {
        if (cachedPage.eTag != null) {
            conn.setRequestProperty("If-None-Match", cachedPage.eTag);
        }
        conn.setIfModifiedSince(cachedPage.pageLastModified);
    }

    conn.connect();

    if (conn.getResponseCode() == HttpURLConnection.HTTP_OK) {

        newCachedPage.eTag = conn.getHeaderField("ETag");
        newCachedPage.pageLastModified = conn.getHeaderFieldDate("Last-Modified", 0);

    } else if (conn.getResponseCode() == HttpURLConnection.HTTP_NOT_MODIFIED) {
        // Never reaches here
    }

I never seem to get the HTTP_NOT_MODIFIED response code, even hitting the same server several times in quick succession - where there is definitely no change to the page. Also, conn.getHeaderField("ETag") always seems to respond null, and sometimes conn.getHeaderFieldDate("Last-Modified", 0) returns 0. I've tried this against a variety of web servers.

Can anyone tell me what I'm doing wrong?

You can only do it successfully if the server supports it. Perhaps it is using `Expires` instead for caching and/or expecting an `If-Not-Modified` request header? Or the server has buggy/poor support for it because the streaming code is written by a starter in a custom servlet? Investigate all response headers of a default HTTP GET request to learn what caching mechanisms the server seems to support. For in-depth background information about HTTP caching mechanisms, read this tutorial: http://www.mnot.net/cache_docs/ — BalusC, Aug 17 '11 at 16:06
The server I'm testing against is Wordpress on a fairly standard LAMP stack. Is it really that rare for servers to support conditional get? — sanity, Aug 17 '11 at 16:40
Uhm, why did you tag `servlets`? Anyway, how about the request headers you obtained? — BalusC, Aug 17 '11 at 16:42
Uhm, because I'm using servlets? I was getting an Expires header, so I've modified my code to support that in addition to conditional get. — sanity, Aug 17 '11 at 16:55
Oh, you're executing this `URLConnection` code inside a servlet method? Sorry, I didn't expect that. I then don't see how that tag is relevant as you would technically have exactly the same problem when doing so in a plain Java class with a `main()` method. Anyway, if you get an `Expires` header, then it just means that you shouldn't request anything until the specified expire time. If you get an `ETag` response header, then it means that you should be able to use `If-None-Match` to test it. If you get a `Last-Modified` response header, you should be able to use `If-Modified-Since` to test it. — BalusC, Aug 17 '11 at 16:57
Ok, I suppose you've got a point, I've removed the servlets tag. Sounds like I'm now supporting the three mechanisms I need to support - just surprised that some of the servers I'm testing against don't seem to take advantage of it. Hey ho. — sanity, Aug 17 '11 at 17:40
Try on `http://cdn3.sstatic.net/stackoverflow/img/favicon.ico` (URL taken from this stackoverflow page request). It gives a `Last-Modified` header of `Wed, 06 Oct 2010 02:53:46 GMT`. If you send a new request with `If-Modified-Since` header of exactly that value, you should get a 304. At least, here it does. — BalusC, Aug 17 '11 at 17:52

BalusC · Accepted Answer · 2011-08-17T18:24:08.400

You're all dependent on the server config.

If you get an Expires response header, then it just means that you don't need to request anything until the specified expire time. If you get a Last-Modified response header, then it means that you should be able to use If-Modified-Since to test it. If you get an ETag response header, then it means that you should be able to use If-None-Match to test it.

Lets take http://cdn3.sstatic.net/stackoverflow/img/favicon.ico as an example (the Stackoverflow's favicon image):

URLConnection connection = new URL("http://cdn3.sstatic.net/stackoverflow/img/favicon.ico").openConnection();
System.out.println(connection.getHeaderFields());

This gives:

{null=[HTTP/1.1 200 OK], ETag=["9d9bd8b1165cb1:0"], Date=[Wed, 17 Aug 2011 17:57:07 GMT], Content-Length=[1150], Last-Modified=[Wed, 06 Oct 2010 02:53:46 GMT], Content-Type=[image/x-icon], Connection=[keep-alive], Accept-Ranges=[bytes], Server=[nginx/0.8.36], X-Cache=[HIT], Cache-Control=[max-age=604800]}

Now, do a If-Modified-Since with the same value as Last-Modified:

URLConnection connection = new URL("http://cdn3.sstatic.net/stackoverflow/img/favicon.ico").openConnection();
connection.setRequestProperty("If-Modified-Since", "Wed, 06 Oct 2010 02:53:46 GMT");
System.out.println(connection.getHeaderFields());

This gives as expected a 304:

{null=[HTTP/1.1 304 Not Modified], ETag=["9d9bd8b1165cb1:0"], Date=[Wed, 17 Aug 2011 17:57:42 GMT], Last-Modified=[Wed, 06 Oct 2010 02:53:46 GMT], Connection=[keep-alive], Server=[nginx/0.8.36], X-Cache=[HIT], Cache-Control=[max-age=604800]}

Now, do a If-None-Match with the same value as ETag:

URLConnection connection = new URL("http://cdn3.sstatic.net/stackoverflow/img/favicon.ico").openConnection();
connection.setRequestProperty("If-None-Match", "9d9bd8b1165cb1:0");
System.out.println(connection.getHeaderFields());

This gives unexpectedly a 200:

{null=[HTTP/1.1 200 OK], ETag=["9d9bd8b1165cb1:0"], Date=[Wed, 17 Aug 2011 18:01:42 GMT], Content-Length=[1150], Last-Modified=[Wed, 06 Oct 2010 02:53:46 GMT], Content-Type=[image/x-icon], Connection=[keep-alive], Accept-Ranges=[bytes], Server=[nginx/0.8.36], X-Cache=[HIT], Cache-Control=[max-age=604800]}

Even more surprising, when the both headers are set with random garbage value as ETag, the server still gives a 304. This is an indication that the If-None-Match is completely ignored by the server behind http://cdn3.sstatic.net. That might be a (proxy) configuration issue or be done fully awarely (not for obvious reasons imho).

Can please help me, I've tried using your method but never returns 304, http://stackoverflow.com/questions/13605425/http-conditional-get. — Moe, Nov 28 '12 at 13:37

score -4 · Answer 2 · answered Mar 17 '12 at 15:40

Or shorter :)

Just try set connection timeout other than 0.

conn.setConnectionTimeout( 3000);

Do it after .openConnection()

final HttpURLConnection conn = (HttpURLConnection) sourceURL.openConnection();
conn.setConnectionTimeout( 3000);

If not set also readTimeout other than 0.

conn.setReadTimeout( 3000);

I'm trying to use Java's HttpURLConnection to do a "conditional get", but I never get a 304 status code

2 Answers2

Linked