Deserialize json one record at a time

Question

I am working with large json files and memory is a concern. I would like to read one object into memory at a time from file. Is this possible?

In ServiceStack.Text docs it says there is an API using reader/stream

But I can't see how to get that working. The files are too large to deserialize in one go. Is it possible to handle this scenario with SS?

Thanks

score 2 · Accepted Answer · answered Jan 08 '20 at 14:51

2

No you'll want to use a streaming JSON parser like System.Text.Json Utf8JsonReader, this is the example on System.Text.Json introductory page:

byte[] data = Encoding.UTF8.GetBytes(json);
Utf8JsonReader reader = new Utf8JsonReader(data, isFinalBlock: true, state: default);

while (reader.Read())
{
    Console.Write(reader.TokenType);

    switch (reader.TokenType)
    {
        case JsonTokenType.PropertyName:
        case JsonTokenType.String:
        {
            string text = reader.GetString();
            Console.Write(" ");
            Console.Write(text);
            break;
        }

        case JsonTokenType.Number:
        {
            int value = reader.GetInt32();
            Console.Write(" ");
            Console.Write(value);
            break;
        }

        // Other token types elided for brevity
    }

    Console.WriteLine();
}

answered Jan 08 '20 at 14:51

mythz

141,670
29
246
390

In case anyone finds this it doesn't seem possible to do this with ` System.Text.Json` as the reader does not accept a byte stream, the byte array must be fully loaded. Newtonsoft JsonTextReader however does accept a stream – Guerrilla Jan 08 '20 at 23:17
@Guerrilla quite the opposite - that's why Utf8JsonReader is used in ASP.NET Core to handle requests (that's why it was built in the first place). It's meant to be used with `System.IO.Pipelines` and `ReadOnlySequence` or `ReadOnlySpan` inputs, *not* byte arrays. The spans and sequences *aren't* arrays, they are a view into the data produced by the pipeline. The entire `System.Text.Json` namespace is built for a very specific scenario, sacrificing ease of use for performance and minimal memory usage – Panagiotis Kanavos Jan 09 '20 at 14:33
Thanks, I am working on my understanding. Can't find any simple examples of deserializing object by object for it – Guerrilla Jan 09 '20 at 14:37
@Guerrilla This is answered [here](https://stackoverflow.com/questions/54983533/parsing-a-json-file-with-net-core-3-0-system-text-json/55429664#55429664). – Panagiotis Kanavos Jan 09 '20 at 14:49

Deserialize json one record at a time

1 Answers1