When streaming a Json file with GSON's JsonReader, can you dump an object to string?

Question

I am working in Java using the GSON library to process a huge JSON document.

I absolutely cannot load this into memory, it is upwards of 8G and will crash the JVM with an OOM if I try to.

So I use JsonReader to process the stream of JSON.

There are certain times when I reach a BEGIN_OBJECT that want to just dump the contents of the object to string instead of processing each individual element.

For example, I am parsing something like this:

[ { "SchemaVersion":"15.0.0.0","LibraryVersion":"16.0.7324.1200","ErrorInfo":{ "ErrorMessage":"Access denied. You do not have permission to perform this action or access this resource.","ErrorValue":null,"TraceCorrelationId":"03ab459e-7076-5000-c1a7-196f1fc54384","ErrorCode":-2147024891,"ErrorTypeName":"System.UnauthorizedAccessException" },"TraceCorrelationId":"03ab459e-7076-5000-c1a7-196f1fc54384" } ]

When I get to ErrorInfo, I just want that as a string instead of parsed out. I don't want to have to parse each piece.

So here is the code:

try (InputStream is = (InputStream)response.getEntity();
           InputStreamReader inputStreamReader = new InputStreamReader(is);
           JsonReader jsonReader = new JsonReader(inputStreamReader)) {
        if (jsonReader.peek() != JsonToken.BEGIN_ARRAY) {
          return;
        }
        jsonReader.beginArray();
        while (jsonReader.hasNext()) {
          jsonReader.beginObject(); // Start of the default object on every request

          jsonReader.skipValue(); // name SchemaVersion
          jsonReader.skipValue(); // value
          jsonReader.skipValue(); // name LibraryVersion
          jsonReader.skipValue(); // value
          jsonReader.skipValue(); // name ErrorInfo
          if (jsonReader.peek() == JsonToken.BEGIN_OBJECT) {
            // I want to dump the upcoming object to string here
...

Is it possible to dump the contents of the json reader for a given begin object until it ends to string?

Similar question, but not quite the same: Get a dump of a section (object) of JSON

Have you tried Apache Spark to read the JSON file? Seems like your JSON has the right format. — Nikhil, May 06 '20 at 14:45
Trying it now! Want to convert that to an answer and get some points? — Nicholas DiPiazza, May 06 '20 at 14:52
I hope you have the link. I have deleted both the link and the answer. I had posted it as an answer to your request. But I was told to delete it. Good luck! — Nikhil, May 06 '20 at 15:27

Nicholas DiPiazza · Accepted Answer · 2020-05-06T16:22:22.080

Special thanks to https://stackoverflow.com/users/3389828/nikhil for the answer in the comments. You can use Gson.fromJson(jsonReader, Map.class) for this exact situation.

Gson gson = new Gson();
try (InputStream is = (InputStream)response.getEntity();
           InputStreamReader inputStreamReader = new InputStreamReader(is);
           JsonReader jsonReader = new JsonReader(inputStreamReader)) {
        if (jsonReader.peek() != JsonToken.BEGIN_ARRAY) {
          return;
        }
        jsonReader.beginArray();
        while (jsonReader.hasNext()) {
          Map header = gson.fromJson(jsonReader, Map.class);
          if (header.get("ErrorInfo") != null) {
            String errorDump = ((Map)header.get("ErrorInfo")).toString();
            // Now I have the error dump I need but I also 
            // didn't load the entire thing into memory!
...

more info: https://javadeveloperzone.com/java-8/java-parse-large-json-file-gson-example/

When streaming a Json file with GSON's JsonReader, can you dump an object to string?

1 Answers1