0

I am trying to implement solution in asp.net core 3. I need to make few private properties REDACTED for security purposes while logging into file. As JsonDocument is readonly and I cannot use JsonPath as well, I have shifted to NewtonSoft for this purpose and use JObject.Parse. My concerns are:

  • How much it will affect my performance?

  • Will it lead to memory leaks?

Code:

var requestTokens = JObject.Parse(requestBodyText); // parse request/response 

foreach (string property in RedactedFields) //for each redacted field in list
{
    JToken token = requestTokens.SelectToken(property); // apply jsonpath
    if (token != null) 
    {
        // jsonpath found
        ((Newtonsoft.Json.Linq.JValue)token).Value = "******REDACTED******";
        requestTokens.SelectToken(property).Replace(token);
    }
}
Muqadar Ali
  • 87
  • 11
  • Please, share the corresponding code and details – Pavel Anikhouski Apr 21 '20 at 08:43
  • @PavelAnikhouski I have updated question with concerned code, please look into it now – Muqadar Ali Apr 21 '20 at 08:48
  • First off, it is very hard to create memory leaks in C# as the garbage collection will take care of that. Unless you yourself keep adding to a static list or do some recursive looping that never stops. And for performance... I'd say run this in a unittest and determine if the performance for both a single run and in bulk is sufficient for your solution. Apart from that it is really hard to help you on this matter. – Davey van Tilburg Apr 21 '20 at 11:22
  • @DaveyvanTilburg I have an old solution that has memory leaks and it goes to 100% RAM regardless of RAM size (16 to 72 GB tested). I am revamping the old code and I have had significant achievement in terms of memory but this piece of code gets me confused. System.text.json has JsonDocument and if we do not dispose it, it may lead to memory leaks. Same could be said for JObject or not? – Muqadar Ali Apr 21 '20 at 11:27
  • Ah I see, really high intensity then haha. Well what I know from the garbage collector is that it does not clean objects from the heap (where all reference objects live) unless there is no new memory available. So... lets say, in very high intensity it maxes out, then cleans up as GC.Collect is called, and the circle repeats. We use JObjects for parsing messages in our message channel that processes more than a few million messages a day. No problems there for us atleast. – Davey van Tilburg Apr 21 '20 at 11:47
  • `JObject` cannot lead to memory leaks because it holds no unmanaged memory or leases on global memory pool memory. But any time you use a string longer than 42,500 characters it will end up on the [large object heap](https://stackoverflow.com/a/8953503/3744182) which can fragment memory leading to increased memory use and/or performance degradation over time. Loading the entire JSON into a single `requestBodyText` is where you are likely to first run into problems. You should load the JSON via streaming instead. – dbc Apr 21 '20 at 15:26
  • Also, load the `JObject` with [`JsonLoadSettings.LineInfoHandling = LineInfoHandling.Ignore`](https://www.newtonsoft.com/json/help/html/P_Newtonsoft_Json_Linq_JsonLoadSettings_LineInfoHandling.htm) to reduce memory overhead. – dbc Apr 21 '20 at 15:27
  • Are the properties you need to make private at the root level or nested deep in the JSON hierarchy? – dbc Apr 21 '20 at 15:50
  • Thanks @dbc, properties can be nested as well as on root level. Different api has different data and request objects. Can you please refer me to any performance document / forum for JsonLoadSettings.LineInfoHandling = LineInfoHandling.Ignore?. Also, 42,500 characters seem too much. we usually do not have that much characters in requests. But this method would be used in hot parts of the code such as log middleware and database/file logging in business logic layers. – Muqadar Ali Apr 21 '20 at 16:12
  • @MuqadarAli - I am the source for recommending `LineInfoHandling.Ignore` to reduce memory use (gold badge in [tag:json.net]). You might also take a look at https://www.newtonsoft.com/json/help/html/Performance.htm – dbc Apr 21 '20 at 16:26
  • Thanks @dbc, this is the only part of my api that uses json.net otherwise I use system.text.json and really needed help in this. – Muqadar Ali Apr 21 '20 at 16:30
  • The only other recommendation I can make from looking at your code is to eliminate `requestBodyText` and deserialize directly to `JObject` or `JObject` from the response stream. I just found the following article which seems on point: https://johnthiriet.com/efficient-api-calls/ – dbc Apr 21 '20 at 16:44
  • @dbc, requestBodyText is string read from PipeReader. Will remember your recommendation and apply load testing ;) – Muqadar Ali Apr 21 '20 at 17:37
  • `PipeReader` has an [`AsStream()`](https://learn.microsoft.com/en-us/dotnet/api/system.io.pipelines.pipereader.asstream?view=dotnet-plat-ext-3.1#System_IO_Pipelines_PipeReader_AsStream_System_Boolean_) which may help here. – dbc Apr 21 '20 at 18:13
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/212194/discussion-between-muqadar-ali-and-dbc). – Muqadar Ali Apr 21 '20 at 18:51

0 Answers0