TokenBuffer Jackson Json

Question

I am wondering what to do about large files of JSON.

I don't want to store them in memory so I don't want to do JsonNodes because I think that stores the entire tree in memory.

My other idea was using TokenBuffer. However, I'm wondering how this works. Does the TokenBuffer store the entire document as well ? Is there a max limit. I know that it's a performance best practice but if I do:

TokenBuffer buff = jParser.readValueAs(TokenBuffer.class);

It seems like it reads the whole document at once (which I don't want).

Answer 1

The purpose of TokenBuffer is to store an extensible array of JSON tokens in memory. It does that by creating at first 1 Segment object with 16 JsonToken objects and then adding new Segment objects as needed.

You are correct to guess that the entire document will be loaded in memory. The only difference is that instead of storing an array of chars it stores the tokens. The performance advantages according to the docs:

1. You can reprocess JSON tokens without re-parsing JSON content from textual representation.
2. It's faster if you want to iterate over all tokens in the order they were appended in the buffer.

TokenBuffer is not a low level buffer of a disk file in memory.

I you only want to parse a file once without loading all of it at once in memory skip the TokenBuffer. Just createJsonParser from a JsonFactory or MappingJsonFactory and get tokens with nextToken. Example.