Out of memory error while parsing a large JSON using Jackson library on Android

Question

I am using Jackson library to parse a large JSON response from server. Size of the json is around 7-8 mb.

I am getting the outOfMemoryError on this piece of code:

ObjectMapper mapper = new ObjectMapper();
JsonNode rootParser = mapper.readValue(is, JsonNode.class);

and this is the exception that I am getting:

    01-14 13:13:20.103: E/AndroidRuntime(25468): FATAL EXCEPTION: Thread-13
    01-14 13:13:20.103: E/AndroidRuntime(25468): java.lang.OutOfMemoryError
    01-14 13:13:20.103: E/AndroidRuntime(25468): at java.util.ArrayList.add(ArrayList.java:123)
01-14 13:13:20.103: E/AndroidRuntime(25468):    at org.codehaus.jackson.node.ArrayNode._add(ArrayNode.java:722)
01-14 13:13:20.103: E/AndroidRuntime(25468):    at org.codehaus.jackson.node.ArrayNode.add(ArrayNode.java:203)
01-14 13:13:20.103: E/AndroidRuntime(25468):    at org.codehaus.jackson.map.deser.std.BaseNodeDeserializer.deserializeArray(JsonNodeDeserializer.java:224)
01-14 13:13:20.103: E/AndroidRuntime(25468):    at org.codehaus.jackson.map.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:200)
01-14 13:13:20.103: E/AndroidRuntime(25468):    at org.codehaus.jackson.map.deser.std.BaseNodeDeserializer.deserializeArray(JsonNodeDeserializer.java:224)
01-14 13:13:20.103: E/AndroidRuntime(25468):    at org.codehaus.jackson.map.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:200)
01-14 13:13:20.103: E/AndroidRuntime(25468):    at org.codehaus.jackson.map.deser.std.BaseNodeDeserializer.deserializeArray(JsonNodeDeserializer.java:224)
01-14 13:13:20.103: E/AndroidRuntime(25468):    at org.codehaus.jackson.map.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:200)
01-14 13:13:20.103: E/AndroidRuntime(25468):    at org.codehaus.jackson.map.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:197)
01-14 13:13:20.103: E/AndroidRuntime(25468):    at org.codehaus.jackson.map.deser.std.BaseNodeDeserializer.deserializeArray(JsonNodeDeserializer.java:224)
01-14 13:13:20.103: E/AndroidRuntime(25468):    at org.codehaus.jackson.map.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:200)
01-14 13:13:20.103: E/AndroidRuntime(25468):    at org.codehaus.jackson.map.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:197)
01-14 13:13:20.103: E/AndroidRuntime(25468):    at org.codehaus.jackson.map.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:58)
01-14 13:13:20.103: E/AndroidRuntime(25468):    at org.codehaus.jackson.map.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:15)
01-14 13:13:20.103: E/AndroidRuntime(25468):    at org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2732)
01-14 13:13:20.103: E/AndroidRuntime(25468):    at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1909)
01-14 13:13:20.103: E/AndroidRuntime(25468):    at com.sarla.smartglance.communication.JsonDecoder.decodeResponse(JsonDecoder.java:87)
01-14 13:13:20.103: E/AndroidRuntime(25468):    at com.sarla.smartglance.communication.JsonDecoder.decode(JsonDecoder.java:68)
01-14 13:13:20.103: E/AndroidRuntime(25468):    at com.sarla.smartglance.communication.MHttpManager$1.run(MHttpManager.java:86)

I tried everything but couldn't find any solution to parse such a large amount of data on android.

Have you tried to just use `.readTree()` instead of reading into a `JsonNode`? — fge, Jan 14 '13 at 17:51
Gson won't help -- tree model is heavy-weight and about same size on different libs. — StaxMan, Jan 14 '13 at 18:18

score 8 · Accepted Answer · answered Jan 14 '13 at 18:24

With 7-8 megs of JSON, tree model that you are using will typically use 20 - 50 megs of memory (dom models are 3-5x as big, both for XML and JSON). There isn't much you can do about that, regardless of library used: they all build trees using Lists and Maps, which is a heavy-weight way of doing it.

Instead you should consider using Plain Old Java Objects (POJOs), which will use much less memory. For this you need to model POJO that matches your JSON structure; without knowing structure I can't give an example (if you add sample on question, I can), but code to parse is similar to GSON code referenced by another answer:

MyValue value = mapper.readValue(json, MyValue.class);

This will work on Jackson as well as many other Java JSON libs (Gson, Genson at least), and will also be faster method to use. JSON trees are inherently expensive and heavy-weight, and not to be used for multi-megatbyte content.

Finally, if your input consists of a sequence of items, there are even better ways to slice it up (which can be done regardless of whether individual items would be JsonNodes or POJOs!). But I don't know if your content is like that.

Still can not understand why people use so much DOM apis...I suppose they don't even work directly on those objects so still need to convert it to POJOs... If only a small part of the json stream is needed, to reduce the data in memory jacksons streaming api could be used. — eugen, Jan 14 '13 at 19:12
Thanks a lot StaxMan.. I did notice that it consumed 20-30 MB of memory but there is nothing much I could do to reduce it. For POJO based model, I have to change the entire structure of parsing. My input is a bit complex as it is an analytical data. So if this is the only way then I guess I have to go with that. — Ramit, Jan 14 '13 at 20:58
Another possibility is that if (and only if) part of model is a sequence (List/Array) of things, you can use incremental parsing also. If so, either need to use `ObjectMapper.readValues()` or raw Streaming API (`JsonParser`) for iteration, then `readValueAs()` when located at root of sub-time. — StaxMan, Jan 15 '13 at 16:48

score 1 · Answer 2 · answered Jan 14 '13 at 17:53

We use here the gson lib, and with the code above we can get files larger than 50Mb without problem:

public static <T extends Object> T readFile(String caminho_arquivo, Type type) {

    GsonBuilder gson_builder = new GsonBuilder();

    final SimpleDateFormat sdf_date     = new SimpleDateFormat("yyyy-MM-dd");
    final SimpleDateFormat sdf_datetime = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.SSS");

    gson_builder.registerTypeAdapter(Date.class, new JsonDeserializer<Date>(){

        @Override
        public Date deserialize(JsonElement json, Type typeOfT, JsonDeserializationContext context) throws JsonParseException {

            try {
                if (json.getAsJsonPrimitive().getAsString().length() == 10)
                    return sdf_date.parse(json.getAsJsonPrimitive().getAsString());
                else
                    return sdf_datetime.parse(json.getAsJsonPrimitive().getAsString());

            } catch (ParseException e) {
                Log.e("JSON", "Erro na deserialização de datas no JSON: " + json.getAsJsonPrimitive().getAsString());
                return null;
            }
        }

    });

    Gson gson = gson_builder.create();

    File fileJSON = new File(caminho_arquivo);

    FileReader reader = null;

    try {
        reader = new FileReader(fileJSON);

        return gson.fromJson(reader, type);

    } catch (FileNotFoundException e) {
        e.printStackTrace();
    } finally {
        try {
            reader.close();

            if (fileJSON.exists())
                fileJSON.delete();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

    return null;
}

Try this lib, that's a good one, with the jackson we use only in the server side, because jackson is more slow in the Android than gson, at least in our test.

Above code is buggy -- does not specify character encoding to use for `FileReader`, so will probably barf on non-ASCII input. But there should be no need to construct a `Reader, just pass in `InputStream`. — StaxMan, Jan 14 '13 at 18:19
Also, links to performance test would be great -- the only commonly cited one (http://www.martinadamek.com/2011/02/04/json-parsers-performance-on-android-with-warmup-and-multiple-iterations/) does not show Gson as faster. Gson does have faster startup time, however; and version 2.1 is ok (earlier versions were very slow). — StaxMan, Jan 14 '13 at 18:20
Well, it's not buggy, it's working very well for me, I had special characters without problem. The "caminho_arquivo" is the location of the file in the SDCard. I use that because I get a zipped file from the server, after that I unzip the file and get the .json file... — William Da Silva, Jan 14 '13 at 18:22
And with Jackson don't help me when I need to parse large files, so for my app GSON was the best, even more slow, can read large files. — William Da Silva, Jan 14 '13 at 18:23
It may work on your platform; but it can easily fail on others; hence it is broken. You should just specify the encoding (UTF-8). This is a common mistake made by Java developers, so perhaps you just cut'n pasted it from somewhere. Was not meant as an insult, although wording may have been strong. — StaxMan, Jan 14 '13 at 18:25
As to Gson vs Jackson: just note that code you gave is different from original one -- you are using POJOs (which makes sense!), but Jackson code referred did not. It may be easily changed to do that, and Jackson approach will likewise work. So this is not a real difference between two approaches. — StaxMan, Jan 14 '13 at 18:28

score 1 · Answer 3 · answered Jan 14 '13 at 18:07

1

Try and use:

JsonNode rootParser = mapper.readTree(is);

instead.

answered Jan 14 '13 at 18:07

fge

119,121
33
254
329

Out of memory error while parsing a large JSON using Jackson library on Android

3 Answers3

Linked