1

I geting data from API in JSON and storing it localy in file using JSON.NET but there is up to 2,5 million records. It works like this :

  • get small amount of data from API and add it to List object
  • save in new file (if file not exists) or OVERRIDE it (if file exists)

My problem is that List of objects must store large amount od data (millions)

My save looks like this

public bool SaveJSON(string path, TData data,Formatting formatting = Formatting.Indented)
{
        bool success = false;
        while(!success)
        {
            try
            {
                using (StreamWriter file = File.CreateText(path))
                {
                    JsonSerializer serializer = new JsonSerializer();
                    serializer.Formatting = Formatting.Indented;
                    serializer.Serialize(file, data);
                    success = true;
                }
            }
            catch(Exception ex)
            {
                Console.WriteLine("Error : " ex.Message);
            }
        }
        return true;

}

With this JSON file looks like

[
   {
     ...
   },
   ...
   {
     ...
   }
]

I know there is File method File.AppendText but with it i get file like this

[
   {
     ...
   },
   ...
   {
     ...
   }
]
[
   {
     ...
   }
]

so non-separated arrays and it can't be deserialized correctly.

How can I deal with this? I thought about removing last line

]

but all solutions I found were about reading whole file to the string and then making changes in it.

Erik Philips
  • 53,428
  • 11
  • 128
  • 150
frappy
  • 21
  • 5
  • I'm not sure what you are trying to do. Are you trying to append/merge some newly received objects to some others already saved to disk file? – Ňɏssa Pøngjǣrdenlarp Dec 15 '17 at 00:08
  • 2
    Why are you trying to append anything to JSON? Deserialize the JSON, modify the object, and Reserialize it. Trying to manually change the json (string/file) is an absolutely terrible idea. – Erik Philips Dec 15 '17 at 00:10
  • Before deserializing, do a `string.Replace("][", "")`. Or write brackets and comma manualy and serialize item by item instead of whole array. To add new item, replace the last `]` by a `,` – Kalten Dec 15 '17 at 00:12
  • Exactly, but I must store in memory whole collection and then save it as new file every time. But I want just append some objects to existing file. – frappy Dec 15 '17 at 00:15
  • And it is good to deserialize the few millions objects..? – frappy Dec 15 '17 at 00:17
  • Rather than "pure" JSON, you might consider switching to [Newline Delimited JSON](http://ndjson.org/) -- basically, a bunch of separate JSON objects concatenated together. See [Line delimited json serializing and de-serializing](https://stackoverflow.com/q/29729063) and [Serialize as NDJSON using Json.NET](https://stackoverflow.com/q/44787652) for how to read and write this format with Json.NET. – dbc Dec 15 '17 at 00:32

0 Answers0