Problems with decoding Stack Exchange API response

Question

I'm trying to receive a response for a regular api call like https://api.stackexchange.com/2.1/me/associated?key=myKey&access_token=myAccessToken

As a response I get something like �� _��l��|[U9��J��b qPmp轢��N�m�x�M��=�A��...

I think everything works okay this far, as the response should be GZIPped. But when trying to decompress the String I get an error: java.io.IOException: unknown format (magic number ef1f) (thrown by GZIPInputStream).

Currently I am using the following class:

import java.io.BufferedReader;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.UnsupportedEncodingException;
import java.util.ArrayList;
import java.util.zip.GZIPInputStream;

import org.apache.http.HttpResponse;
import org.apache.http.StatusLine;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.DefaultHttpClient;

import android.os.AsyncTask;
import android.util.JsonReader;
import android.util.Log;

public class RetrieveAccounts extends AsyncTask<Void, String, String> {

    protected String myFeed = "";
    String retrieveAddress = "https://api.stackexchange.com/2.1/me/associated?key=myKey&access_token=";

    public RetrieveAccounts(String accessToken) {
        retrieveAddress += accessToken;
        Log.i("URL", "Retrieve Address: " + retrieveAddress);
    }

    public ArrayList<String> readMessagesArray(JsonReader reader) throws IOException {
        ArrayList<String> messages = new ArrayList<String>();

        reader.beginArray();
        while (reader.hasNext()) {
            if(reader.nextName().equals("site_name")) {
                messages.add(reader.nextString());
            }
        }
        reader.endArray();
        return messages;
    }

    @Override
    protected String doInBackground(Void... params) {
        Log.i("URL", "doInBackground");
        StringBuilder builder = new StringBuilder();
        HttpClient client = new DefaultHttpClient();
        HttpGet httpGet = new HttpGet(retrieveAddress);
        httpGet.setHeader("Accept-Encoding", "GZIP");
        for(int i = 0; i < httpGet.getAllHeaders().length; i++) {
            Log.i("URL", "Header " + i + ": " + httpGet.getAllHeaders()[i].getValue());
        }
        try {
            HttpResponse response = client.execute(httpGet);
            StatusLine statusLine = response.getStatusLine();
            int statusCode = statusLine.getStatusCode();

            if(statusCode == 200) {
                BufferedReader reader = new BufferedReader(new InputStreamReader(response.getEntity().getContent()));

                String line;
                while ((line = reader.readLine()) != null) {
                    builder.append(line);
                }
            }
            else {
                Log.e("URL", "Failed to download file");
            }
        }
        catch (ClientProtocolException e) {
            e.printStackTrace();
        }
        catch (IOException e) {
            e.printStackTrace();
        }
        return builder.toString();
    }

    @Override
    protected void onPostExecute(String result) {       
        String meinErgebnis = "";
        try {
            meinErgebnis = decompress(result.getBytes());
        }
        catch (UnsupportedEncodingException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
        catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
        Log.i("URL", "Ergebnis: " + meinErgebnis);
    }

    public static String decompress(byte[] bytes) throws UnsupportedEncodingException, IOException {
//        if (str == null || str.length() == 0) {
//            return str;
//        }
        System.out.println("Input String length : " + bytes.length);
        GZIPInputStream gis = new GZIPInputStream(new ByteArrayInputStream(bytes));
        BufferedReader bf = new BufferedReader(new InputStreamReader(gis, "UTF-8"));
        String outStr = "";
        String line;
        while ((line=bf.readLine())!=null) {
          outStr += line;
 }
        System.out.println("Output String lenght : " + outStr.length());
        return outStr;
     }

}

Stack Trace for IOException:

[dalvik.system.VMStack.getThreadStackTrace(Native Method),
java.lang.Thread.getStackTrace(Thread.java:591),
de.stefansurkamp.stacknotifications.RetrieveAccounts.decompress(RetrieveAccounts.java:111),
de.stefansurkamp.stacknotifications.RetrieveAccounts.onPostExecute(RetrieveAccounts.java:89),
de.stefansurkamp.stacknotifications.RetrieveAccounts.onPostExecute(RetrieveAccounts.java:1), 
android.os.AsyncTask.finish(AsyncTask.java:602),
android.os.AsyncTask.access$600(AsyncTask.java:156),
android.os.AsyncTask$InternalHandler.handleMessage(AsyncTask.java:615), 
android.os.Handler.dispatchMessage(Handler.java:99), 
android.os.Looper.loop(Looper.java:137),
android.app.ActivityThread.main(ActivityThread.java:4493),
java.lang.reflect.Method.invokeNative(Native Method),
java.lang.reflect.Method.invoke(Method.java:511), 
com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:788),
com.android.internal.os.ZygoteInit.main(ZygoteInit.java:555), 
dalvik.system.NativeStart.main(Native Method)]

I think these are the Hex values for the response:

1F 8B 08 00 00 00 00 00 04 00 BD 94 4D 6F DC 20 10 86 FF 8A C5 D9 4A 00 63 3E 7C 8B AA 3D F4 D4 4A 51 4F 55 65 11 2F EB 45 B5 C1 05 9C 6C BB DA FF 5E EC DD 36 A9 BB 4E 9C AD 52 DF 3C 33 C0 BC 0F 2F B3 07 3A A8 D6 83 E2 F3 1E DC C9 75 AD CA CA F6 26 C4 C0 1E D4 B6 59 83 02 A6 C0 EB E6 5E 39 50 64 29 B8 73 D6 FC 50 A0 40 D9 61 88 07 55 1A D9 C6 7F 70 1B 64 F5 35 F9 10 EB 36 8D 7D 00 A7 64 EF 9A 98 DB 86 D0 15 D7 D7 7E 28 B1 A7 8A AB CA B6 B1 AA F7 CA 95 3A 1E 23 38 86 98 A6 C0 A9 AE 0F 32 68 6B 40 91 A3 3C 05 B2 1A 3B 3A 16 E5 94 33 96 82 CA A9 B1 A4 5C CB 30 36 83 18 17 8C 67 22 05 8D F4 A1 8C 6B 94 F7 BF B3 9C 53 4A 10 1E 36 33 FE 21 1E 38 EE 18 F5 44 6D DF 7A E5 C7 AD 4E 31 44 0F E9 02 14 F0 09 8A 29 89 BE 53 2E F9 14 85 9D A7 30 A4 07 D9 53 02 98 22 82 C9 9F 04 10 44 0B 09 70 44 A0 C0 88 CE 10 18 F8 40 04 A7 04 CE 00 80 AF D5 8F CF 3A E1 A6 EB FC BC 0B 64 CC FE A5 9F 10 98 5F 2C 9F 0A 46 31 46 B3 06 18 DC 95 2D 90 8F FE 55 FE 70 F1 C9 6A 17 2F 59 2B 53 A9 E4 88 63 B5 AB B6 D2 D4 EA 2C 92 7E 77 35 52 51 A7 A2 29 99 D8 3F 61 13 32 68 39 99 5C C0 9C CE 3D 0D C6 51 CE 33 FE 16 64 E0 84 CC 47 67 6B 27 DB 56 39 BF 84 4A F7 58 FE 3C 1E 04 33 41 E8 C5 CE 81 9C 63 32 CB 27 FA 4A 70 B8 C4 39 AF 7E 38 53 3E 37 66 ED AC 5E 27 2B 13 B6 BD D7 B1 9B 45 9C E4 71 D9 0B 16 12 22 BF F8 71 31 86 45 84 94 CD 22 E2 19 A6 EC 7F 20 7A 6F 36 D6 B5 63 77 C9 AD AA 7A A7 C3 F7 25 8C FC A9 F6 79 48 98 23 CC 2F 86 94 11 96 C3 B9 09 F4 98 7D 7B 48 EF 6C 6D 74 D0 F7 71 F6 54 E3 0C 5A 64 A3 CA D6 BE D2 2F B9 88 4C 4C 24 C4 F2 31 C4 A2 85 F2 B9 31 84 10 A7 64 89 87 D0 E1 CB 10 B4 41 96 4E B5 52 1B 6D EA A1 0F 4E 7E 85 5B B9 1B 2E 2E 7E 29 D8 4A 5F B6 D6 C5 43 36 B2 F1 EA F0 13 51 B5 01 C0 F0 08 00 00

Any ideas on how to solve this problem?

Can you include a full stack trace? Also, verify that you can hit the api url with your browser (and the tokens filled in). Next, please include your code with the GZIPInputStream (or at least check the response encoding). — Elliott Frisch, Dec 27 '13 at 23:07
@ElliottFrisch: I added the stack trace to my original post. Yes, i can hit the API with my browser, everything works fine this way. The response encoding is 'utf-8'... — Stefan Surkamp, Dec 28 '13 at 10:35
That is a valid gzip stream. It starts with the correct magic number, `1f 8b`. That data must not be making it to the gzip decompressor unscathed, since it is reporting `1f ef` as the first two bytes. You should make sure that there is no character-set interpretation going on somewhere. The bytes should be treated as bytes. — Mark Adler, Jan 02 '14 at 16:11

score 0 · Answer 1 · answered Dec 26 '15 at 17:53

As you are using GZIP compression on data sent through a response, you should first use a Base64 encoder on the compressed data and then send it. On the receiving side you should decode with Base64 and then decompress.

import org.apache.commons.codec.binary.Base64;
...
String base64Data = Base64.encodeBase64String(compressedData);

Apache Commons Codec is a library you could use to encode/decode Base64.

Be careful, if you are developing for Android, as in Android 2.2 the library is already included (v1.4 I think).

Problems with decoding Stack Exchange API response

1 Answers1