GSON: Java out of memory error heap space

Question

I'm trying to convert a large object into json format as a string using the below code,I'm getting out of memory exception.However we are using heap size of 1GB.

gson.toJson(this);

How can we use gson streaming API(https://sites.google.com/site/gson/streaming) to eliminate this error without increasing heap size ?

 Caused by: java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Arrays.java:2367)
        at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)
        at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)
        at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415)
        at java.lang.StringBuffer.append(StringBuffer.java:237)
        at java.io.StringWriter.write(StringWriter.java:112)
        at com.google.gson.stream.JsonWriter.string(JsonWriter.java:576)
        at com.google.gson.stream.JsonWriter.writeDeferredName(JsonWriter.java:402)
        at com.google.gson.stream.JsonWriter.value(JsonWriter.java:417)
        at com.google.gson.internal.bind.TypeAdapters$16.write(TypeAdapters.java:426)
        at com.google.gson.internal.bind.TypeAdapters$16.write(TypeAdapters.java:410)
        at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:68)
        at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.write(MapTypeAdapterFactory.java:208)
        at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.write(MapTypeAdapterFactory.java:145)
        at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:68)
        at com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.write(CollectionTypeAdapterFactory.java:97)
        at com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.write(CollectionTypeAdapterFactory.java:61)
        at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:68)
        at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.write(ReflectiveTypeAdapterFactory.java:112)
        at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.write(ReflectiveTypeAdapterFactory.java:239)
        at com.google.gson.Gson.toJson(Gson.java:661)
        at com.google.gson.Gson.toJson(Gson.java:640)
        at com.google.gson.Gson.toJson(Gson.java:595)
        at com.google.gson.Gson.toJson(Gson.java:575)

How bis is the object you're talking about? if the object's memory footprint is close to 1G, this might not be solvable with a different serialization approach. — f1sh, Oct 23 '17 at 11:02
Object has some properties and an arraylist which has 104219 records,each record has 5 properties. — Coded9, Oct 23 '17 at 11:04
@Ashok Are you sure you would like the result as a single big json? I recently converted ~50GB of data into json files, but I partitioned the data to multiple files, each were just a few MBs. — szab.kel, Oct 23 '17 at 11:06
In order to get a full fledged example, please provide your *JSON* document structure and what you have already tried. — tmarwen, Oct 23 '17 at 11:07
@tmarwen please find the structure here http://jsoneditoronline.org/?id=67b451d0ec82b7e6ece4a1d768566012,the data property contains more than 100000 records.I have already mentioned the code I tried --> gson.toJson(this); — Coded9, Oct 23 '17 at 11:24
Dear downvoter,if you know the answer please let me know rather than downvoting ! — Coded9, Oct 23 '17 at 15:57
I've encountered the same error. Can you tell what version of GSON you are using? My stacktrace is similar: at java.lang.OutOfMemoryError.()V (OutOfMemoryError.java:48) at java.util.Arrays.copyOf([CI)[C (Arrays.java:3332) at java.lang.AbstractStringBuilder.ensureCapacityInternal(I)V (AbstractStringBuilder.java:124) ... at com.google.gson.stream.JsonWriter.value(Ljava/lang/Number;)Lcom/google/gson/stream/JsonWriter; (JsonWriter.java:534) — Richard Sand, Feb 23 '18 at 21:09

score 2 · Answer 1 · answered Oct 23 '17 at 10:56

2

You can read big Json files without loading the whole thing into memory, by using a streaming approach.

Even Gson supports Streaming: https://google.github.io/gson/apidocs/com/google/gson/stream/JsonReader.html

Also: JAVA - Best approach to parse huge (extra large) JSON file

answered Oct 23 '17 at 10:56

szab.kel

2,356
5
40
74

I'm actually trying to write to json format as a string – Coded9 Oct 23 '17 at 11:00
Oh, true, sorry. But the Stream API has Write mode too, and If you are already using the Gson Stream API to achieve this, I don't know what you would like to hear as an answer, because you did not post any additional code. – szab.kel Oct 23 '17 at 11:05
well on desktop it still requires a lot of memory even when using JsonReader (which expects input stream as input), e.g. I need to get some info from the next json file `instances_train2017.json` (448MB), you can find it in http://images.cocodataset.org/annotations/annotations_trainval2017.zip So on my PC with 6GB memory I get `memory out` exception – user155 Mar 07 '19 at 11:47

GSON: Java out of memory error heap space

1 Answers1