GZIPInputStream - Corrupt GZIP Trailer

Question

I have a static helper method responsible for getting a compressed JSON string from our Rails application, and decompressing the data before returning a String representation.

I wrote two JUnit tests that, one that tests that the JSON parses correctly, and another more basic test that determines whether a string of length greater than zero is returned from the server.

The problem: When I run the test suite, the first test method succeeds properly, and the other fails with an IOException and the message "Corrupt GZIP trailer" (see code below). I have determined that it is not the test itself that is failing, as the "successful" test is reversed when I cause the tests to run in opposite order (in other words, no matter what, it is always the second test that fails, no matter which of the two tests is run second).

This is the helper method:

public static String doHTTPGet(String urlString) throws IOException{
    URL weatherAPI = new URL(urlString);
    HttpURLConnection apiConnection = (HttpURLConnection) weatherAPI.openConnection();
    apiConnection.setRequestMethod("GET");
    apiConnection.setRequestProperty("Accept-Encoding", "gzip");

    apiConnection.connect();

    BufferedInputStream bufferedInputStream = new BufferedInputStream(apiConnection.getInputStream());
    byte[] inputByteBuffer = new byte[10 * 1024];
    ByteArrayOutputStream outputStream = new ByteArrayOutputStream(10 * 1024); // initialize the output stream with at least one buffer's worth of bytes
    while(bufferedInputStream.read(inputByteBuffer)  > -1){
        outputStream.write(inputByteBuffer);
    }

    outputStream.close();
    bufferedInputStream.close();
    apiConnection.disconnect();

    ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(outputStream.toByteArray());
    byteArrayInputStream.close();

    GZIPInputStream gis = new GZIPInputStream(byteArrayInputStream);
    InputStreamReader inputStreamReader = new InputStreamReader(gis, "UTF-8");
    BufferedReader reader = new BufferedReader(inputStreamReader);

    String decompressedResponse = "";
    String line;

    // readLine() is generating the IOException on the second pass.
    while((line = reader.readLine()) != null){
        decompressedResponse += line;
    }

    reader.close();
    inputStreamReader.close();
    gis.close();

    return decompressedResponse;
}

The error occurs towards the bottom of the helper method, on the line while((line = reader.readLine()) != null).... Specifically, the error occurs on the reader.readLine().

And the two test methods:

@Test
public void testHttpGet(){
    try {
        // FILTERED_API_URL_WITH_TOKEN is merely the URL with an auth token
        String apiResponse = HTTPHelper.doHTTPGet(GlobalConstants.FILTERED_API_URL_WITH_TOKEN);
        assertNotNull(apiResponse);
        assertTrue("The size of the API response should be greater than zero. It is an empty string.", apiResponse.length() > 0);
    } catch (IOException e) {
        e.printStackTrace();
        assertTrue("An exception occured while trying to perform the HTTP Get to the api at URL " + GlobalConstants.FILTERED_API_URL_WITH_TOKEN, false);
    }
}

@Test
public void testAPIContent(){
    try {
        // the getAPIJson() method basically does the same as the testHttpGet
        // method, but converts the string to a json
        JSONObject jsonObject = XMLProducerFromAPI.getAPIJson();
        System.out.println(jsonObject);
        assertNotNull(jsonObject);
    } catch (IOException e) {
        e.printStackTrace();
        assertTrue("An IOException occured. See stack trace", false);
    } catch (JSONException e) {
        e.printStackTrace();
        assertTrue("A JSONException occured. See stack trace", false);
    }
}

I have read through this question and the answer, but I don't believe its applicable, (or maybe it is and I misunderstood, let me know if that is the case), and I tried their approach and only received the same message.

As the doHTTPGet method is static, and the objects created are done so within the body of the method, nothing (streams, connection objects, etc) should be reused. Frankly, I'm stumped.

Question: Have I done something wrong in my helper code, or have I misunderstood some usage of some object, that would yield the the "Corrupt GZIP Trailer" message? In short, what would cause this error in my scenario?

As always, please let me know if I've left anything out of this question.

EDIT

This is the stack trace:

java.io.IOException: Corrupt GZIP trailer
    at java.util.zip.GZIPInputStream.readTrailer(GZIPInputStream.java:200)
    at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:92)
    at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
    at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
    at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
    at java.io.InputStreamReader.read(InputStreamReader.java:167)
    at java.io.BufferedReader.fill(BufferedReader.java:136)
    at java.io.BufferedReader.readLine(BufferedReader.java:299)
    at java.io.BufferedReader.readLine(BufferedReader.java:362)
    at com.weathertx.xmlserver.support.HTTPHelper.doHTTPGet(HTTPHelper.java:60)
    at com.weathertx.xmlserver.tests.HttpHelperTest.getAPIResponse(HttpHelperTest.java:47)
    at com.weathertx.xmlserver.tests.HttpHelperTest.testHttpGet(HttpHelperTest.java:21)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
    at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
    at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)

Also, sadly, the API in question is restricted by IP, so unfortunately I cannot give access without knowing everybody's IP. — Paul Richter, Oct 12 '13 at 01:42

score 0 · Accepted Answer · edited May 23 '17 at 11:59

0

Problem has been solved. To be frank, I don't quite understand why it would not work originally, or what was wrong with it (other than apparently being overly and unnecessarily complicated). Thanks to this solution, which I somehow missed my first search through, I was able to solve the problem by basically implementing more or less exactly what they did. This is my final code:

public static String doHTTPGet(String urlString) throws IOException{
    URL weatherAPI = new URL(urlString);
    HttpURLConnection apiConnection = (HttpURLConnection) weatherAPI.openConnection();
    apiConnection.setRequestMethod("GET");
    apiConnection.setRequestProperty("Accept-Encoding", "gzip");

    apiConnection.connect();

    InputStream gzippedResponse = apiConnection.getInputStream();
    InputStream decompressedResponse = new GZIPInputStream(gzippedResponse);
    Reader reader = new InputStreamReader(decompressedResponse, "UTF-8");
    StringWriter writer = new StringWriter();

    char[] buffer = new char[10240];
    for(int length = 0; (length = reader.read(buffer)) > 0;){
        writer.write(buffer, 0, length);
    }

    writer.close();
    reader.close();
    decompressedResponse.close();
    gzippedResponse.close();
    apiConnection.disconnect();

    return writer.toString();
}

So ultimately, I did not need to pass the data through byte array streams and all over the place. Other than my original approach being convoluted, if anybody knows specifically why my original algorithm yielded the "Corrupted GZIP trailer" error message after the first call of this method, by all means let me know.

edited May 23 '17 at 11:59

Community

1
1

answered Oct 18 '13 at 19:52

Paul Richter

10,908
10
52
85

1

Compare the write() calls. This one is correct, using the read count. Your original doesn't, – user207421 Oct 18 '13 at 20:05
@EJP So my original would constantly write exactly 10240 bytes each time, and this one writes however many it could read, up to 10240, meaning the last read might be a number of bytes less than that (correct?). But, after the streams and connections and such are all closed, when the method is called a second time, new streams and connections are opened. Is there left over garbage trickling through, or something to that effect? – Paul Richter Oct 18 '13 at 20:14
You are writing garbage on the last write, and you are getting a message that says exactly that. It's perfectly clear. – user207421 Oct 18 '13 at 20:28
@EJP Yup, I understand that now. That's not quite what I was asking though; specifically, I'm curious why the failure only happens on the **second** call to this method. Both calls are exactly the same, they point to the same URL and receive the same compressed data. Given that all the objects are local to this method, and the ones that are streams are closed before the end of the method, it seems odd to me. While I understand what you've said, there's still the missing answer that explains why it succeeds the first time, the fails the second, consistently. Anyways, thank you for your help. – Paul Richter Oct 18 '13 at 20:56
Is the second call with the same URL? If it's a different URL and the first download is a multiple of your buffer size, the problem won't occur. – user207421 Oct 18 '13 at 21:16

GZIPInputStream - Corrupt GZIP Trailer

1 Answers1

Linked