3

Does the .NET provided System.Text.Json namespace provide an analogous service/class to Newtonsoft.Json's JsonReader/JsonTextReader?

Our new app has been built using only System.Text.Json dependencies but we now have a requirement to process extremely large JSON files (10's of GBs). This require us to read the files as a Stream sequentially as opposed to loading the entire objects into memory.

We were able to hook this up using Newtonsoft.Json's JsonReader/JsonTextReader but was wondering if this same functionality existed without using this external dependency and mixing Json libraries.

using (StreamReader sr = new(stream))
using (JsonReader jsonReader = new JsonTextReader(sr))
{
    while (jsonReader.Read())
    {
        // Do Work
    }
}
smk081
  • 783
  • 1
  • 10
  • 36
  • 3
    Basically no. If your root JSON container is an array you can use [`JsonSerializer.DeserializeAsyncEnumerable()`](https://learn.microsoft.com/en-us/dotnet/api/system.text.json.jsonserializer.deserializeasyncenumerable) to iterate through the root array items without loading the entire thing into memory. Otherwise, for workarounds see [Parsing a JSON file with .NET core 3.0/System.text.Json](https://stackoverflow.com/q/54983533/3744182). – dbc Apr 13 '23 at 12:44
  • Seems to be by [this article](https://visualstudiomagazine.com/articles/2020/07/28/json-serializers.aspx) that System is better in performance to Newtonsoft, but I'm not so sure, you should make a profiling of this before a conversion. I like and prefeer Newtonsoft anyway. – Leandro Bardelli Apr 13 '23 at 12:44
  • Yeah, unfortunately our JSON payloads are not structured this way - they are essentially documents that wrap around some large nested arrays of objects but have some top-level metadata. We saw that DeserializeAsyncEnumerable() but could find a way to index into the JSON object (without reading it all into memory) or seek in the Stream somehow to the property that contained the array to make that approach work. – smk081 Apr 13 '23 at 12:59
  • 1
    @smk081 - MSFT prioritized async + pipeline support over straightforward iterative streaming, and the result is that what you need to do is quite nontrivial. Newtonsoft's not going anywhere, I would stick with it if it works and is the right tool for the job (Heck there is one situation where `JsonReaderWriterFactory` is the right tool to use.) The only JSON serializer you should not use is `JavaScriptSerializer`, it truly has been deprecated and isn't even ported to .NET Core. – dbc Apr 13 '23 at 16:04
  • Thanks for the background! Couldn't find this feature on the roadmap for .NET 8, so I'm guessing this isn't getting in there anytime soon. Using Newtonsoft for this one module and System.Text.Json everywhere else isn't terrible in the short term though. I appreciate the info! – smk081 Apr 13 '23 at 17:26

0 Answers0