22

I have an ASP.NET MVC website that gets about 6500 hits a day, on a shared hosting platform at Server Intellect. I keep seeing app restarts in the logs and I cannot figure out why.

I've read Scott Gu's article here: http://weblogs.asp.net/scottgu/archive/2005/12/14/433194.aspx and implemented the technique, and here's what shows up in my log:

Application Shutdown: 
_shutDownMessage=HostingEnvironment initiated shutdown 
HostingEnvironment caused shutdown    
_shutDownStack=at
System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)     at 
System.Environment.get_StackTrace()     at 
System.Web.Hosting.HostingEnvironment.InitiateShutdownInternal()     at 
System.Web.Hosting.HostingEnvironment.InitiateShutdown()     at 
System.Web.Hosting.PipelineRuntime.StopProcessing()

It seems to occur about every five minutes.

Are there any other ways to debug this?

UPDATE: Here are the application pool settings mentioned by Softion:

CPU

  • Limit : 0
  • Limit Action : no action
  • Limit Interval : 5 Minutes

Process Model

  • Idle Timeout : 20 Minutes
  • Ping Maximum Response Time : 90 Seconds
  • Startup Time Limit : 90 Seconds

Rapid-Fail Protection

  • Enabled : True
  • Failure Interval : 5 Minutes

Recycling

  • Private Memory Limit : 100 MB
  • Regular Time Interval : 1740 Minutes (29 Hours)
  • Request Limit : 0
  • Specific Times : none
  • Virtual Memory Limit : 0
Dave Thieben
  • 5,388
  • 2
  • 28
  • 38
  • What I've found to diagnose the issue is to attach an event handler to AppDomain.Current.FirstChanceException. You'll then be notified of all exceptions, caught and uncaught which might help you find the root cause. – Razor Mar 17 '17 at 11:57

4 Answers4

30

You can easily grab the reason of the shutdown by HostingEnvironment. You read Scott Gu article, but you missed its comments.

     var shutdownReason = HostingEnvironment.ShutdownReason;

Shutdown reason codes

If the reason is HostingEnvironment, check the IIS application pool parameters controlling recycling. I've put a red dot near each one. Check the description in the bottom help box in your own copy for full info.

You can ask your provider to give you the applicationHost.config file where all these parameters are set. They find it in C:\Windows\System32\inetsrv\config. I'm sure you can also get them using some .NET api.

enter image description here

For 6500 hits a day, which is a very low hit rate, i'm betting the "Idle time-out" is set to 5mn.

Update (moved comments to here //jgauffin)

  • CPU Limit 0 = disabled.
  • Process Model Idle Timeout : 20 Minutes (20mn without a request recycles your app).
  • Rapid-Fail Protection enabled (5mn). You need to know the maximum failures count. If your app throws more than this exception count in 5mn it will we recycled.
  • Private Memory Limit : 100 MB. Yes you should profile, this is a low limit.
  • Regular Time Interval : 1740 Minutes (29 Hours): it will recycle every 29h.
  • Request Limit : 0 (disabled).
  • Virtual Memory Limit : 0 (disabled).
  • Rapid-Fail Protection enabled (5mn). You need the maximum failures count. If your app throws more than this exception count in 5mn it recycles. If it recycles every 5mn this should be the thing to check. There should be 0 unhandled exception in secondary worker threads. Wrap your code into a try catch there.
jgauffin
  • 99,844
  • 45
  • 235
  • 372
Softlion
  • 12,281
  • 11
  • 58
  • 88
  • I updated the question with the settings reported by the host. I have a feeling the Private Memory Limit may be my problem. I'll profile the app locally to investigate. – Dave Thieben Jun 07 '11 at 13:02
  • 1
    Check your worker threads including those created using ThreadPool.QueueUserWorkItem. If an exception in a user thread is not catched it will crash the thread, the main threads will remain live but the rapid fail protection will still count the exception. As it checks the exception count every 5mn, if the count is over the "max failures count" the app pool will be restarted. – Softlion Jun 09 '11 at 23:06
  • I've upgraded my account to raise the private memory limit from 100Mb to 200Mb and I've had only 1 app restart since (yesterday morning). I've also seen a drastic decrease in exceptions thrown due to untimely app shutdowns. Thank you so much for your help. – Dave Thieben Jun 12 '11 at 14:39
  • Where should i have to put this piece of code. Like if in any event or what – Beingnin May 31 '19 at 07:21
3

re update:

The settings asked to the provider help, but is way better to ask for information on the reason of the restarts like I mentioned on my original answer i.e. the actual log entries of the restarts like I mentioned on my orig answer. From those you can know specifically what was triggered, I've seen happen one hitting different limits.

You really have to:

profile your application with a realistic amount of test data

My money is on hitting resource limits set by your hosting provider.

Before going crazy with optimization without a target, contact your provider and ask them to give you information on the restarts.

Typical recycles:

  • idle x amount of time / like 15 mins
  • more than x amount of memory / like 200 MB
  • more than x % processor over y time / like 70 over 1 minute
  • a daily recycle

Once you know the case, you have to find out what's taking those resources. For this you have to profile your application with a realistic amount of test data. Knowing if it is memory or processor can help on knowing what to look for.

eglasius
  • 35,831
  • 5
  • 65
  • 110
1

Is IIS set to recycle the app pool frequently?

Is there some kind of runaway memory leak in the app pool?

joelt
  • 2,672
  • 2
  • 24
  • 32
  • If you look at the application pool properties in IIS, you can see if it's set to recycle frequently. Memory-wise, I think the place I'd start is just watching performance monitor. – joelt Mar 17 '11 at 01:26
  • I guess I should have mentioned this is on a shared-hosting platform at Server Intellect. – Dave Thieben Mar 17 '11 at 15:45
  • That creates a challenge. Asking them for support might be the best bet. If you have a development machine, you could also try to replicate those conditions and see if you get a matching error response. The IIS setting is easy to replicate. A memory leak is probably harder...you could try just creating a huge array... – joelt Mar 18 '11 at 23:17
1

It requires a bit of know how on what your app does here's a list of things that can cause the app to restart/reset or even shut down

  • StackOverflowException
  • OutOfMemoryException
  • Any unhandled exception that crashes a thread
  • CodeContracts use Environment.FailFast when a contract violation occurs

Exceptions are quite easy to track if you can reproduce the issue with a debugger attached you can go into Visual Studio and enable all exceptions when they are thrown not caught by user code. It will sometimes reveal intresting stuff that otherwise is hidden away.

John Leidegren
  • 59,920
  • 20
  • 131
  • 152