Serialize large Java Object to Json Strings

Question

I have a large Java object (100MB+) that needs to be serialized to Json string. I was using jackson Json until I found out it reaches memory limit on large object:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOfRange(Arrays.java:3664)
    at java.lang.String.<init>(String.java:207)
    at java.lang.StringBuilder.toString(StringBuilder.java:407)
    at com.fasterxml.jackson.core.util.TextBuffer.contentsAsString(TextBuffer.java:349)
    at com.fasterxml.jackson.core.io.SegmentedStringWriter.getAndClear(SegmentedStringWriter.java:83)
    at com.fasterxml.jackson.databind.ObjectWriter.writeValueAsString(ObjectWriter.java:682)
    at JavaTest.objToJson(JavaTest.java:76)

The method that does the conversion looks like this:

public static String objToJson(Object obj) {

    ObjectWriter ow = new ObjectMapper().writer().withDefaultPrettyPrinter();
    String json = null;

    try {
        json = ow.writeValueAsString(obj);
    } catch (JsonProcessingException e) {
        System.out.println("Error in objToJson(): " + e);
        e.getMessage();
        e.printStackTrace();
    }
    return json;
}

The environment is memory intense so that I won't be able to arbitrarily increase the heap size.

Any suggestions on more memory efficient ways of serializing Java objects? I found jackson library can do streaming but many posts say about the other way around, i.e., deserializing json string to object. Also, it seems jackson lib has better performance than other packages such as gson, which I haven't tried personally.

Thanks.

In what situation would you want to transfer *100MB+* of data to another system? — Scary Wombat, Apr 27 '17 at 04:50
check this http://stackoverflow.com/questions/31477711/streaming-json-directly-to-response-with-jackson — Sachin Gupta, Apr 27 '17 at 04:53
Is the rest of the System not java? It seems really intensive to convert such large data to JSON, when you could maybe SOAP. RMI or serialization of the Object could be better — Scary Wombat, Apr 27 '17 at 05:01
Possible duplicate of [Streaming JSON directly to response with Jackson](http://stackoverflow.com/questions/31477711/streaming-json-directly-to-response-with-jackson) — Lyubomyr Shaydariv, Apr 27 '17 at 08:05
@SachinGupta that seems to work and consume less memory, thanks. I'll update my solution and memory consumption in answer section. — Shiyu, Apr 27 '17 at 15:31
@LyubomyrShaydariv I'm also expecting memory consumption information like answer below, you link (which is same as Sachin Gupta gave btw) doesn't have that information — Shiyu, Apr 27 '17 at 15:42
@Shiyu The linked question is a clear hint to use streaming. You have a JVM killer: `writeValueAsString` -- this is the root of all evil. — Lyubomyr Shaydariv, Apr 27 '17 at 15:47
@ScaryWombat unfortunately in my scenario, I have to do the json serialization. It's not un-common to ser/deser large objects not only for data sharing across different parties, but also for storage (and binary is not an option neither) — Shiyu, Apr 27 '17 at 15:48
@LyubomyrShaydariv As shown in my stacktrace, I can see that too. In addition to pointing out the root cause, showing some working examples (that link shows examples on the web API side rather) and memory consumption estimation would be more helpful. Thanks for the help. — Shiyu, Apr 27 '17 at 15:50

score 0 · Accepted Answer · edited May 23 '17 at 12:02

This answer contains useful hint of using Jackson Streaming API. Some more Google searches show useful examples here and here.

For memory consumption, I created a simple program that generate a ~33MB in memory object and here's the rough estimate:

using standard json conversion in the Question, it uses up to ~150MB memory
using jackson streaming API, it uses up to ~60MB

However, I do need to create the JsonGenerator manually, which is highly dependent on the object structure and it's not scalable; also for complex object, creating the JsonGenerator is not quite easy.

Serialize large Java Object to Json Strings

1 Answers1