Parsing JSON with both streaming and DOM access?

Question

I have a JSON document that looks like this:

It's a collection of arrays - findings, assets, assetGroups, etc. I wrote a function that takes the filename and the requested array name, and returns an ArrayList<> of the array entries as strings (which I re-parse to JSON on the client side). This works great when the files are smaller, but this one file is over 1.6GB in size so I blow out memory if I try and instantiate it all as a JSONObject. I want to try Jackson or GSon streaming APIs, but I'm getting wrapped around the axle trying to mix streaming APIs with direct DOM access. Like, I want to stream the JSON until I reach the "assetGroups" node, then iterate over that array and return the List<> of its contents. Does that make sense? Any help?

Answer 1

(The following is for Gson)

You would probably have to start with a JsonReader to parse the JSON document in a streaming way. You would then first use its beginObject() method to enter the top level JSON object. Afterwards in a loop guarded by a hasNext() check you would obtain the next member name with nextName() and compare it with the desired name. If the name is not the one you are interested in, you can call skipValue() to skip its value:

JsonReader jsonReader = new JsonReader(reader);
jsonReader.beginObject();

while (jsonReader.hasNext()) {
    String name = jsonReader.nextName();

    if (name.equals(desiredName)) {

        ... // extract the data; see next section of this answer
        
        // Alternatively you could also return here already, ignoring the remainder
        // of the JSON data
        
    } else {
        jsonReader.skipValue();
    }
}

jsonReader.endObject();

if (jsonReader.peek() != JsonToken.END_DOCUMENT) {
    throw new IllegalArgumentException("Trailing data after JSON value");
}

You might also want to add checks to verify that the desired member actually exists in the JSON document, and to verify that it only exists exactly once and not multiple times.

If you only want to write back a subset of the JSON document without performing any content modifications to it, then you don't need to parse it into DOM objects. Instead you could directly read from the JsonReader and write to a JsonWriter:

static void transferTo(JsonReader reader, JsonWriter writer) throws IOException {
    NumberHolder numberHolder = new NumberHolder();
    int nestingDepth = 0;

    while (true) {
        JsonToken token = reader.peek();
        switch (token) {
            case BEGIN_ARRAY:
                reader.beginArray();
                writer.beginArray();
                nestingDepth++;
                break;
            case END_ARRAY:
                reader.endArray();
                writer.endArray();

                nestingDepth--;
                if (nestingDepth <= 0) {
                    return;
                }

                break;
            case BEGIN_OBJECT:
                reader.beginObject();
                writer.beginObject();
                nestingDepth++;
                break;
            case NAME:
                writer.name(reader.nextName());
                break;
            case END_OBJECT:
                reader.endObject();
                writer.endObject();

                nestingDepth--;
                if (nestingDepth <= 0) {
                    return;
                }

                break;
            case BOOLEAN:
                writer.value(reader.nextBoolean());
                break;
            case NULL:
                reader.nextNull();
                writer.nullValue();
                break;
            case NUMBER:
                // Read the number as string
                String numberAsString = reader.nextString();

                // Slightly hacky workaround to preserve the original number value
                // without having to parse it (which could lead to precision loss)
                numberHolder.value = numberAsString;
                writer.value(numberHolder);

                break;
            case STRING:
                writer.value(reader.nextString());
                break;
            case END_DOCUMENT:
                throw new IllegalStateException("Unexpected end of document");
            default:
                throw new AssertionError("Unknown JSON token: " + token);
        }
    }
}

You would call this method at the location marked with "extract the data" in the first code snippet.

If instead you do need the relevant JSON document section as DOM, then you can first use Gson.getAdapter to obtain the adapter, for example for your List<...> or for JsonArray (the generic DOM class; here it is less likely that you risk any precision loss during conversion). And then you can use read(JsonReader) of that adapter at the location marked with "extract the data" in the first code snippet.
It would not recommend directly using Gson.fromJson(JsonReader, ...) because it changes the configuration of the given JsonReader; unfortunately this pitfall is not properly documented at the moment (Gson issue).

Note that both approaches do not preserve the original JSON formatting, but content-wise that should not make a difference.