Reading data from URL returning strange characters

Question

I am trying to grab the data from a json file through java. If I navigate to the URL using my browser, everything displays fine, but if I try to get the data using java I get get a bunch of characters that cannot be interpreted or parsed. Note that this code works with other JSON Files. Could this be a server side thing with the way the JSON file is created? I tried messing around with different character sets and that did not seem to fix the problem.

public static void main(String[] args) throws Exception {
    URL url = new URL("http://www.minecraftpvp.com/api/ping.json");
    URLConnection connection = url.openConnection();
    BufferedReader in = new BufferedReader(new InputStreamReader(connection.getInputStream()));
    boolean hasLine = true;
    while (hasLine) {
        String line = in.readLine();
        if (line != null) {
            System.out.println(line);
        } else {
            hasLine = false;
        }
    }
}

The output I get from this is just a ton of strange characters that make no sense at all. Where if I change the url to something like google.com, it works fine.

EDIT: JSON URL from StackExchange API returning jibberish? Seemed to have answered my question. I tried searching before I asked to make sure the answer wasn't here and couldn't find anything. Guess I didn't look hard enough.

Answer 1

Yes that URL is returning gzip encoded content by default.

You can do one of three things:

1. Explicitly set the Accept-Encoding: header in your request. A web service should not return gzip compression unless it is listed as an accepted encoding in the request, so this website is not being very friendly. Your browser is setting it as accepted I suspect, that is why you can see it there. Just set it to an empty value and it should as per the spec return non-encoded responses, your mileage may vary on this one.

2. Or use the answer in this How to handle non-UTF8 html page in Java? that shows how to decompress the response. This should be the preferred option over #1.

3. And/or Ask the person hosting the service to implement the recommended scheme which is to only provide compressed responses if the client says it can handle them or if it can infer it from the browser fingerprint with high confidence.

Best of luck C.

Answer 2

You need to inspect the Content-Encoding header. The URL in question improperly returns gzip-compressed content even when you don't ask for it, and you'll need to run it through a decoder.