Take 1gb ram to parse json object and give System.OutOfMemoryException after performing any other filter

Question

public void ReadJsonFile()
{
    try
    {
        string json = string.Empty;
        using (StreamReader r = new StreamReader(val))
        {
            json = r.ReadToEnd();
            var test = JObject.Parse(json);
            JArray items = (JArray)test["locations"];
            int length = items.Count;

            data = new List<Info>();
            for (int i = 0; i < items.Count; i++)
            {
                var d = test["locations"][i]["timestampMs"];
                double dTimeSpan = Convert.ToDouble(d);
                DateTime dtReturn = new DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc).AddSeconds(Math.Round(dTimeSpan / 1000d)).ToLocalTime();
                string printDate = dtReturn.DayOfWeek.ToString() + "," + " " + dtReturn.ToShortDateString() + " " + dtReturn.ToShortTimeString();
                day = dtReturn.DayOfWeek.ToString();
                date = dtReturn.ToShortDateString();
                time = dtReturn.ToShortTimeString();
                var e = test["locations"][i]["latitudeE7"];
                var f = test["locations"][i]["longitudeE7"];
                var n = test["locations"][i]["accuracy"];
                accuracy = n.ToString();
               // getLocationByGeoLocation(e.ToString(), f.ToString());                   
                var g = test["locations"][i]["activity"] != null;
                if (g == true)
                {
                    JArray items1 = (JArray)test["locations"][i]["activity"];
                    int length1 = items1.Count;
                    while (j < items1.Count)
                    {
                        if (j == 0)
                        {
                            var h = test["locations"][i]["activity"][j]["activity"][j]["type"];
                            type = h.ToString();
                            j = 1;
                        }
                        else { }
                        j++;
                    }
                    j = 0;
                }
                else { }

                Info ddm = new Info(day, date, time, lat, longi, address, accuracy, type);
                data.Add(ddm);
                type = "";
            }
        }
        return;
    }
    catch (Exception ex)
    {
        MessageBox.Show(ex.Message);
    }
}

I am trying to parse JSON file. val is the name of my file to parse. Using StreamReader I am reading each line when I am trying to parse using jObject I will take around 1gb memory and give me System.OutOfMemoryException how can I parse JObject using small memory. Please help me with this I don't have much idea of JSON.

Firstly, get rid of `json = r.ReadToEnd()` and stream directly from the file using [`JObject.Load(JsonReader)`](https://www.newtonsoft.com/json/help/html/M_Newtonsoft_Json_Linq_JObject_Load.htm). You're allocating a gigantic string unnecessarily. Then see where you stand. — dbc, Dec 18 '18 at 04:57
Like @dbc mentioned for a big file, instead of allocating everything in one go, stream from the file — Hasan Emrah Süngü, Dec 18 '18 at 04:59
Next, why are you loading to an intermediate `JToken` hierarchy at all? Why not deserialize directly to an appropriate data model? Switching to direct deserialization will save lots of intermediate memory. — dbc, Dec 18 '18 at 05:01
Finally, if you still can't load the file using deserialization, you'll need to adopt a piecemeal stream technique like the one shown in [Parsing large json file in .NET](https://stackoverflow.com/q/32227436) and [How to parse huge JSON file as stream in Json.NET?](https://stackoverflow.com/q/43747477/3744182). But before doing that, first get rid of the intermediate objects `json` and `test`. — dbc, Dec 18 '18 at 05:02
I don't have much idea about JSON file I take this code from the net but I work for a small file. — user10753256, Dec 18 '18 at 05:03
@user10753256 Could you please past an example of json data you are trying to parse? — Hasan Emrah Süngü, Dec 18 '18 at 05:06
{ "locations" : [ { "timestampMs" : "1521958808759", "latitudeE7" : 196590799, "longitudeE7" : 727169204, "accuracy" : 2299 }, { "timestampMs" : "1521956972681", "latitudeE7" : 196617374, "longitudeE7" : 727147037, "accuracy" : 2400 },} — user10753256, Dec 18 '18 at 05:24

score 2 · Accepted Answer · answered Dec 18 '18 at 05:35

2

Please read about JSON thoroughly. NewtonSof.JSON is are very famous library and it is well documented. Let us get back to your problem. As mentioned in comments you have lots of unnecessary middle steps while trying to parse your file. Moreover, you are trying to parse a big file in one go!. First thing first, this is the layout for your JSON

public partial class Data
{
    [JsonProperty("locations")]
    public Location[] Locations { get; set; }
}

public partial class Location
{
    [JsonProperty("timestampMs")]
    public string TimestampMs { get; set; }

    [JsonProperty("latitudeE7")]
    public long LatitudeE7 { get; set; }

    [JsonProperty("longitudeE7")]
    public long LongitudeE7 { get; set; }

    [JsonProperty("accuracy")]
    public long Accuracy { get; set; }
}

And while you are deserilazing you should do it object by object, not all at once The following assumes that your stream is made of Data type of objects, if it is, made up of Location type of objects you have to change it

using (StreamReader streamReader = new StreamReader(val))
using (JsonTextReader reader = new JsonTextReader(streamReader))
{
    reader.SupportMultipleContent = true;

    var serializer = new JsonSerializer();
    while (reader.Read())
    {
        if (reader.TokenType == JsonToken.StartObject)
        {
            var data = serializer.Deserialize<Data>(reader);
            //data.locations etc etc..
        }
    }
}

answered Dec 18 '18 at 05:35

Hasan Emrah Süngü

3,488
1
15
33

It works fine but how can I apply JArray items = (JArray)test["locations"]; int length = items.Count;and get value from loop through – user10753256 Dec 18 '18 at 06:13
Do you want to get the size of the Locations array – Hasan Emrah Süngü Dec 18 '18 at 06:16
Yes because I am converting timestamp and all four things so use loop for that like in my previous code JArray items = (JArray)test["locations"]; int length = items.Count; – user10753256 Dec 18 '18 at 06:19
Then please use `data.Locations.Length`. Please let me know if that works for you :) – Hasan Emrah Süngü Dec 18 '18 at 06:25
Ya, thank you so much it works. – user10753256 Dec 18 '18 at 06:27
How can i get this value var h = test["locations"][i]["activity"][j]["activity"][j]["type"]; because now every time i get 0 as a value o type – user10753256 Dec 18 '18 at 11:02
You don't need to use i or j as indexes. Look at the class properties then you can access them – Hasan Emrah Süngü Dec 18 '18 at 11:42
"timestampMs" : "1521949958568", "latitudeE7" : 196616755, "longitudeE7" : 727194779, "accuracy" : 2496, "activity" : [ { "timestampMs" : "1521949958783", "activity" : [ { "type" : "STILL", "confidence" : 100 } ] } ] I want this type from activity how can i get using your code – user10753256 Dec 19 '18 at 04:30

score 0 · Answer 2 · answered Dec 18 '18 at 05:03

I could fix the System.OutOfMemoryException with the following steps:

If you are not using Visual Studio Hosting Process:

Uncheck the option:

Project->Properties->Debug->Enable the Visual Studio Hosting Process

if still the problem remains:

Go to Project->Properties->Build Events->Post-Build Event Command line and paste this 2 lines

call "$(DevEnvDir)..\..\vc\vcvarsall.bat" x86
"$(DevEnvDir)..\..\vc\bin\EditBin.exe" "$(TargetPath)"  /LARGEADDRESSAWARE

Now, build the project

Take 1gb ram to parse json object and give System.OutOfMemoryException after performing any other filter

2 Answers2