0

I have come across an issue with our ASP.NET application where occasionally a nightly recycle cases the w3wp to hang.

This is what happens:

Recycle is triggered. obviously this forces ThreadAbortException on all running threads. However, it doesn't seem to trigger a new w3wp, or it is the new w3wp that actually throws the exception (haven't been able to reproduce it yet).

In my logs I get a lot of ThreadAbortException, and the thread count just goes up and up indefinitely, meaning that any new request spawns a new thread that never finishes. If this would have been the old w3wp, any new request would be routed to the newly started w3wp.

Neither the shutdown timeout nor the rapid-fail protection seems to trigger as well, leaving the saite unavailable until it is manually recycled. Most of the time it also eats up a lot of CPU leaving the server almost unusable as well.

We are using Monorail MVC which probably doesn't have anything to do with it, however we do utilize their RescueController system. If we we're to unintentionally catch the ThreadAbortException in our error handling, could that cause a infinite loop that would hang the w3wp so badly that IIS cant recover from it?

BenMorel
  • 34,448
  • 50
  • 182
  • 322
jishi
  • 24,126
  • 6
  • 49
  • 75
  • Might want to make sure you're logging all the recycle events. From powershell: `set-webconfigurationproperty /system.applicationHost/applicationPools/applicationPoolDefaults/recycling -name:logEventOnRecycle -value:"Time, Requests, Schedule, Memory, IsapiUnhealthy, OnDemand, ConfigChange, PrivateMemory"` – Geoffrey McGrath Aug 16 '11 at 01:46

2 Answers2

1

IIS is unable to start up a new worker process successfully due to a resource constraint, likely due to not enough free memory.

Try adjusting your Private Memory Limit in IIS, decreasing the number of Maximum Worker Processes aka Web Garden (rarely set it to greater than 1--if it's set higher than that there is likely an underlying problem to solve), increasing the physical RAM available on your servers, calling Marshal.ReleaseComObject appropriately, and otherwise sorting out what is preventing freeing of memory. You might even consider changing the garbage collection mode from Server to Workstation (see http://msdn.microsoft.com/en-us/library/ms229357.aspx).

Geoffrey McGrath
  • 1,663
  • 1
  • 14
  • 35
  • Thank you very much for the input. We aren't using web garden, so that shouldn't be a problem. Limited memory allocation sounds plausible, I'm gonna look into it. Regarding server garbage collection mode, wasn't aware that there where two different modes. This is a dual quad-core machine (8 in total) but it's running multiple sites. What is the pros and cons for the two modes? It sounds like concurrent garbage collection is a good thing? – jishi Aug 16 '11 at 09:22
  • Before jumping into toggling garbage collections modes, be sure and profile your application to better characterise the problem you're seeing. Run Performance Monitor ("%windir%\system32\perfmon.msc /s"), read this article to help know what performance counters to watch. Tools such as CLRProfiler (http://msdn.microsoft.com/en-us/library/ff650691.aspx) may help. If the evidence then continues to point to memory and garbage collection, see this OS thread: http://stackoverflow.com/questions/4931309/how-to-make-net-garbage-collection-less-frequent – Geoffrey McGrath Aug 16 '11 at 14:13
  • My main problem with this is that it is only in production environment I have noticed this behaviour, and I don't have access to those servers. Looking at the memory graphs that we have for the servers, I don't see any direct drop in available memory during time of recycle, although they monitor it in 5 min intervals. However, the server is usually down approx 30 minutes until a manual reboot is done, so during that time it should be very low I assume, according to the graphs I still have about 3GB available (out of 16GB) – jishi Aug 16 '11 at 14:32
  • You may be able have the operations people attach a debugger that can capture .dmp files and exception logs. If your application isn't too poorly behaved, the overhead might be acceptible (from 5 to 30 % of CPU). See the question I posted (and answered myself tee hee!) http://stackoverflow.com/questions/6618105/how-to-use-adplus-to-only-get-minidumponsecond, perhaps it will help you too. – Geoffrey McGrath Aug 17 '11 at 06:27
0

It turns out that we had an recursive try catch loop for Exception, that catches ThreadAbortException and invokes itself because of inheritance, so it became an infinite recursion.

We had the catch for Exception because we wanted logging and some error handling, and would probably be fine in all other aspects except that ThreadAbortException will keep on throwing during the execution.

jishi
  • 24,126
  • 6
  • 49
  • 75
  • what was it then? – Jason Coyne Sep 06 '16 at 13:48
  • @JasonCoyne I don't think we ever got to the bottom of it. We did some startup optimizations to lessen the startup load we were seeing, and also eventually switched to Win2012 which might have improved things. – jishi Sep 06 '16 at 14:15