Use Jackson To Stream Parse an Array of Json Objects

Question

I have a file that contains a json array of objects:

[ { "test1": "abc" }, { "test2": [1, 2, 3] } ]

I wish to use use Jackson's JsonParser to take an inputstream from this file, and at every call to .next(), I want it to return an object from the array until it runs out of objects or fails.

Is this possible?

Use case: I have a large file with a json array filled with a large number of objects with varying schemas. I want to get one object at a time to avoid loading everything into memory.

EDIT:

I completely forgot to mention. My input is a string that is added to over time. It slowly accumulates json over time. I was hoping to be able to parse it object by object removing the parsed object from the string.

But I suppose that doesn't matter! I can do this manually so long as the jsonParser will return the index into the string.

Answer 1

Yes, you can achieve this sort of part-streaming-part-tree-model processing style using an ObjectMapper:

ObjectMapper mapper = new ObjectMapper();
JsonParser parser = mapper.getFactory().createParser(new File(...));
if(parser.nextToken() != JsonToken.START_ARRAY) {
  throw new IllegalStateException("Expected an array");
}
while(parser.nextToken() == JsonToken.START_OBJECT) {
  // read everything from this START_OBJECT to the matching END_OBJECT
  // and return it as a tree model ObjectNode
  ObjectNode node = mapper.readTree(parser);

  // do whatever you need to do with this object
}

parser.close();

Answer 2

What you are looking for is called Jackson Streaming API. Here is a code snippet using Jackson Streaming API that could help you to achieve what you need.

JsonFactory factory = new JsonFactory();
JsonParser parser = factory.createJsonParser(new File(yourPathToFile));

JsonToken token = parser.nextToken();
if (token == null) {
    // return or throw exception
}

// the first token is supposed to be the start of array '['
if (!JsonToken.START_ARRAY.equals(token)) {
    // return or throw exception
}

// iterate through the content of the array
while (true) {

    token = parser.nextToken();
    if (!JsonToken.START_OBJECT.equals(token)) {
        break;
    }
    if (token == null) {
        break;
    }

    // parse your objects by means of parser.getXxxValue() and/or other parser's methods

}

Answer 3

This example reads custom objects directly from a stream:

source is a java.io.File

ObjectMapper mapper = new ObjectMapper();
JsonParser parser = mapper.getFactory().createParser( source );
if ( parser.nextToken() != JsonToken.START_ARRAY ) {
    throw new Exception( "no array" );
}
while ( parser.nextToken() == JsonToken.START_OBJECT ) {
    CustomObj custom = mapper.readValue( parser, CustomObj.class );
    System.out.println( "" + custom );
}

Answer 4

This is a late answer that builds on Ian Roberts' answer. You can also use a JsonPointer to find the start position if it is nested into a document. This avoids custom coding the slightly cumbersome streaming token approach to get to the start point. In this case, the basePath is "/", but it can be any path that JsonPointer understands.

Path sourceFile = Paths.get("/path/to/my/file.json");
// Point the basePath to a starting point in the file
JsonPointer basePath = JsonPointer.compile("/");
ObjectMapper mapper = new ObjectMapper();
try (InputStream inputSource = Files.newInputStream(sourceFile);
     JsonParser baseParser = mapper.getFactory().createParser(inputSource);
     JsonParser filteredParser = new FilteringParserDelegate(baseParser,
                    new JsonPointerBasedFilter(basePath), false, false);) {
    // Call nextToken once to initialize the filteredParser
    JsonToken basePathToken = filteredParser.nextToken();
    if (basePathToken != JsonToken.START_ARRAY) {
        throw new IllegalStateException("Base path did not point to an array: found " 
                                       + basePathToken);
    }
    while (filteredParser.nextToken() == JsonToken.START_OBJECT) {
        // Parse each object inside of the array into a separate tree model 
        // to keep a fixed memory footprint when parsing files 
        // larger than the available memory
        JsonNode nextNode = mapper.readTree(filteredParser);
        // Consume/process the node for example:
        JsonPointer fieldRelativePath = JsonPointer.compile("/test1");
        JsonNode valueNode = nextNode.at(fieldRelativePath);
        if (!valueNode.isValueNode()) {
            throw new IllegalStateException("Did not find value at "
                    + fieldRelativePath.toString() 
                    + " after setting base to " + basePath.toString());
        }
        System.out.println(valueNode.asText());
    }
}