0

We have an a medium-big asp.net project on dotnet 6. Newtonsoft.JSON and efcore 5 are used.
It's hosted on linux containers governed by k8s on aws.
We're seeing the nodes dying out once in a day or two due to memory issues, they have 4GB RAM available.
When doing memory profiling I've noticed a strange accumulation of instances of RuntimeTypeHelper, GenericFieldInfo, and RuntimeMethodHandler in Generation 2, hundreds of thousands of them after just running locally and navigating thought ~10-20 pages of the application several times over and over. The pages are data-rich, yes, but anyway a steady linear increase in the retained objects count is suspicious.

dotMemory shows they are retained through ServiceProvider enter image description here and sampling couple of them, they have some pretty obscure creation stack trace enter image description here

I've tried migrating everything to dotnet 7, updating all the libraries and switching to System.Text.Json, that didn't noticeably change the overall picture.

What could be the reason for such objects accumulation? Or conversely, is this normal and I should rather focus on memory pressure and look into decreasing allocations elsewhere?

alpha-mouse
  • 4,953
  • 24
  • 36
  • 1
    Have you looked at articles like https://learn.microsoft.com/en-us/dotnet/core/diagnostics/debug-memory-leak https://www.tessferrandez.com/ Many "leaks" are caused by static collections (like caches of things) that are never (or rarely) cleared/purged. – Simon Mourier Jun 07 '23 at 09:56
  • @SimonMourier structurally it's indeed this `LIst` growing over time deep inside `ServiceProvider`. It doesn't look like some issue inside any particular service, but rather something with `ServiceProvider` itself, and I'm struggling to understand why would `ServiceProvider` be doing that exactly, and how to fix that, unfortunately. – alpha-mouse Jun 08 '23 at 08:14
  • It looks like you are creating run-time types on the fly over and over again using reflection.emit and serializing them with Json.NET. The problem here is that Json.NET permanently caches metadata about types, see [Does Json.NET cache types' serialization information?](https://stackoverflow.com/a/33558665/3744182). You can reclaim **some** of that by creating and discarding your own `DefaultContractResolver`, but not all -- some info about enums and data contracts is cached globally. – dbc Jun 10 '23 at 19:32
  • So your options would appear to be 1) Switch to System.Text.Json which doesn't do that. 2) Cache and reuse the dynamic types rather than creating them over and over again. In that case the Json.NET memory use will stabilize. – dbc Jun 10 '23 at 19:35
  • Incidentally I believe Entity Framework Dynamic Proxies are implemented using Reflection.Emit, see [Proxies using Records with a base class broken since .NET 6 #26602](https://github.com/dotnet/efcore/issues/26602). So you may be able to resolve the problem by disabling dynamic proxies, or mapping the proxies to DTOs before serializing with Json.NET. – dbc Jun 10 '23 at 19:36
  • @dbc thanks for the suggestion, will look into the dynamic proxies thing. And will have a closer look at switching to System.Text.Json, maybe I did something wrong then experimenting with switching to it, where I didn't see much difference. – alpha-mouse Jun 10 '23 at 19:44
  • Hmm, well System.Text.Json might be caching things also, its API is less open than Json.NET. – dbc Jun 10 '23 at 20:01

0 Answers0