Okay this one has me scratching my head.
I have an API returning a response body that is a JSON string.
The the data is such that it will continue to grow every week. That is I have a single JSON response that is going to potentially grow forever until it either: 1) consumes the world as we know it - or 2) someone on the API side learns about pagination.
So how do I serialize this if I don't know the size?
I'm thinking it has to be done in chunks, as I have already figured out how to chunk the response to file, but how do I serialize in chunks?
This is really a concept problem so responses in any language will work as long as they can get at the heart of the problem.
I've so far tried stream reading character by character in C# to test for JSON artifacts ({},[] -etc) but the performance was glacial. About 20min to read 1000kb.
I feel like i'm reinventing the wheel here...
EDIT: In Response to @stdunbar 's link.
I've just tried using the solution described in the link with JSON.Net's JsonReader class. A couple problems.
1). There is never any info on JSON.NET's documentation about what a "JSON token" is. Is it the node name and the node value? Is it just the node name?
2). If my data set is potentially infinite, that means my node values are potentially infinite. For example there is a node in my JSON response containing a list. That list grows as the data does. So if I load that list using JsonReader it might just be 1TB some day and wreck my server.
