0

We have a web application that works as a reverse proxy (forwarding requests and responses). This is part of a bigger platform so quite a large number of calls are getting through there. The application is run on IIS and we use ASP.NET WebApis ApiController to receive and HttpClient to resend the calls (newer NuGet packages on .NET 4.6.1). We have disabled IIS worker process recycling.

In production we can see that over about one week the memory usage is constantly growing and then remaining at a high level. Occasionally we are getting OutOfMemoryExceptions. Using R# dotMemory profiler we couldn't find a memory leak but we can see that there is a huge fragmentation (up to 99%!) of Gen 0 to 2 objects. We assume this is related to the OverlappedData objects that dotMemory marks as pinned.

There are some posts out there dealing with this kind of issue, a good one being http://www.ahuwanya.net/blog/post/Buffer-Pooling-for-NET-Socket-Operations - the summary being that the fragmentation is related to the buffer pinning for the IO operations. So on this ground we have some questions:

  1. How could we introduce a buffer pool as shown in the post with Sockets for use with ApiController and HttpClient?
  2. How could we force pinned memory to be released, for example by closing a connection - without having a significant performance loss?
  3. How can we ensure the assumption is correct by getting more detailed information about the pinned memory? We found something about logging stacktraces with Perfmon but a more detailed guide would be appreciated.

Of course any other suggestions, hints or remarks that help us to narrow down and solve the issue are appreciated as well. Please note that we want to solve the issue and not have a workaround for it (such as restarting the IIS worker process every day).

srudin
  • 261
  • 2
  • 13
  • Use static HttpClient – Tony Jun 14 '18 at 02:32
  • See https://stackoverflow.com/questions/27732546/httpclienthandler-httpclient-memory-leak –  Jun 14 '18 at 02:39
  • Why do you have to pin in the first place? – PepitoSh Jun 14 '18 at 02:58
  • @Tony: We already use a static HttpClient. – srudin Jun 14 '18 at 06:24
  • @Amy: As said to Tony we already use a static HttpClient. Therefore this solution does not apply. – srudin Jun 14 '18 at 06:25
  • @PepitoSh: We don't pin - this is done by the respective .NET framework classes. See the provided link in the post for details. – srudin Jun 14 '18 at 06:26
  • Referenced document says just above conclusion: "If using the pattern presented above, it’s important to remember to dispose the buffer when closing the socket. You must explicitly dispose the buffer. It does not define a finalizer, so once a buffer is created, it stays allocated until explicitly disposed." To me this means that you do not free memory explicitly or have an awful lot of sockets open. – PepitoSh Jun 14 '18 at 06:31
  • @PepitoSh Yes, our conclusion points in the same direction. But we do dispose everything we can and as said we were not able to identify a memory leak yet. And due to the nature of our application it is not uncommon to have a number of open connections at any given time. That's what got us to asking the three questions above. – srudin Jun 14 '18 at 10:28
  • In our applications we pre-allocate and re-use a properly sized pool of sufficiently large buffers. These buffers do not need to be pinned - in our applications. They need to be pinned in unsafe or in managed context, but only during actual use. If you manage your buffers on a low level that needs fixed memory locations, I recommend allocating from the large object heap and manage the buffers with your own code... but this is not so .net-like. – PepitoSh Jun 14 '18 at 12:32
  • @PepitoSh We are in an all managed environment thus there is no need to pin anything. We have NO "fixed" statement or unmanaged calls in our entire solution. According to the information provided the pinning seems to be done by the framework when I/O calls are made as these calls invoke unmanaged code. We don't want to manage the buffers ourselves - this is the suggestion provided in the linked information. Our question is how to achieve that in the WebApi context. – srudin Jun 15 '18 at 00:33
  • ***In our current release we don't have this problem anymore*** So please remove your question because there is no issues as explained in question & was not have a proper answer. – Sinto Dec 11 '18 at 05:58
  • @Sinto I disagree. This problem is real and the assumption that the "volatile" might cause it could be very helpful to others. – srudin Dec 11 '18 at 09:33

1 Answers1

0

In our current release we don't have this problem anymore. We have no real explanation for it - the only thing that seems to have changed is that we removed some "volatile" definitions. Could this have been the cause? Maybe yes - in fact we assume it is, but have no real proof for it. According to the documentation it is somewhat unclear whether volatile will prevent memory optimization, but from understanding it is likely. So we just accept it like this...

srudin
  • 261
  • 2
  • 13