0

I have 175MB json file, and I am loading this file data into a static JObject variable to use for a query. However, while running console application, I could see that this process occupy 2.5GB memory.

Here is my code snippet:

public class LocationData
{
    public static JObject LOCATION { get; set; }

    public static void FillData()
    {
        JArray jArray = new JArray();

        using (StreamReader file = File.OpenText("somefiles.json"))
        using (JsonTextReader reader = new JsonTextReader(file))
        {
            jArray.Add(JObject.ReadFrom(reader));
        }

        LocationData.LOCATION = new JObject(new JProperty("Location", jArray));
        jArray = null;
    }
}

I fill LOCATION once thereafter here is my GetData code for querying data from LOCATION:

 LocationData.LOCATION.SelectTokens(tokenString).ToList();

I wanted a solution around this to handle such data and keep data in memory (Since I wanted to use this data frequently) with adequate memory utilization. I request your help on best practices to follow here to manage such data in a C# application. Do let me know in case if you need more info to answer this.

Thanks in advance.

Ramjee Gupta
  • 41
  • 1
  • 1
  • 6
  • 1
    I would find a way to break up the JSON before reading it as a `JObject`. That's a lot of data regardless of how you process it. – Dan Wilson Aug 22 '18 at 13:58
  • If you have the possibility of editing your input json file, rather than having it as one big array, devide it to several arrays, all you need to do is to pick your division points, and add the closing ], and start [ at those points. Then you can read in each array one by one. ALso see this post : https://stackoverflow.com/questions/32227436/parsing-large-json-file-in-net – PiJei Aug 22 '18 at 14:07
  • @PiJei: I have already broken json files into multiple pieces adding in jArray. However, altogether file weight is 75MB. – Ramjee Gupta Aug 22 '18 at 14:27
  • What if you convert your file to a sql table and have an object as a record there? This will need some work, but might be worth it. – PiJei Aug 22 '18 at 14:31
  • @PiJei: I wanted to use this dataset very frequently hence using static variable. You can suggest any other way. I dont want to interact with sql for frequent data queries – Ramjee Gupta Aug 22 '18 at 14:35
  • So your problem right now is that you are occupying lots of memory right? but you never crash or get any errors am I right? – PiJei Aug 22 '18 at 14:38
  • It just occured to me that you can specify your json deserialize settings such to ignore null or default fields, is that something that could reduce the size in your case? – PiJei Aug 22 '18 at 14:40
  • @PiJei: correct. I am looking for a solution on how to use such large datasets in application which needs to available for frequent queries. – Ramjee Gupta Aug 22 '18 at 14:41
  • Possible duplicate of [Parsing large json file in .NET](https://stackoverflow.com/questions/32227436/parsing-large-json-file-in-net) – Liam Aug 22 '18 at 14:42
  • And this json file will not change right? you have it once and all your queries will use it? you could also edit your file to rename the keys to shorter names etc to cut it short. – PiJei Aug 22 '18 at 14:43
  • @PiJei: yes, json files will not change. does reducing keys will have significant impact? not sure, but I'll try this. BDW, I have only 4-5 fields. – Ramjee Gupta Aug 22 '18 at 14:55
  • If you dont want to use sql like approaches, you could use some key value storage like mongo db maybe? and if none of these you need to design your own system, for example break this file into several files with some indexing capability. For example if you have dates you know that dates x to y are in file 1, and when you get a query with this range you know which file to load to memeory and so on.. but again you need a storage system, – PiJei Aug 22 '18 at 14:59
  • @PiJei: Sure, I have thought of this. Just wanted to put here if I can get any good ideas other than this. Thank you for your inputs. – Ramjee Gupta Aug 22 '18 at 15:31
  • Yes, that is what I can think of. Good luck. – PiJei Aug 22 '18 at 15:31
  • Regardless of the JSON part, consider the memory footprint of the final class, having a large number of objects in-memory will for sure consume quite a lot of RAM, even without considering the overhead of parsing beforehand. – Alejandro Aug 22 '18 at 15:42

0 Answers0