Should Application_End fire on an automatic App Pool Recycle?

Question

I have read this, this, this and this plus a dozen other posts/blogs.

I have an ASP.Net app in shared hosting that is frequently recycling. We use NLog and have the following code in global.asax

void Application_Start(object sender, EventArgs e) 
{
    NLog.Logger logger = NLog.LogManager.GetCurrentClassLogger();
    logger.Debug("\r\n\r\nAPPLICATION STARTING\r\n\r\n");
}
protected void Application_OnEnd(Object sender, EventArgs e)
{
    NLog.Logger logger = NLog.LogManager.GetCurrentClassLogger();
    logger.Debug("\r\n\r\nAPPLICATION_OnEnd\r\n\r\n");
}

void Application_End(object sender, EventArgs e) 
{
        HttpRuntime runtime = (HttpRuntime)typeof(System.Web.HttpRuntime).InvokeMember("_theRuntime", BindingFlags.NonPublic  | BindingFlags.Static  | BindingFlags.GetField,  null,  null,  null);

if (runtime == null)
    return;

string shutDownMessage = (string)runtime.GetType().InvokeMember("_shutDownMessage",  BindingFlags.NonPublic  | BindingFlags.Instance  | BindingFlags.GetField,  null,  runtime,  null);

string shutDownStack = (string)runtime.GetType().InvokeMember("_shutDownStack",   BindingFlags.NonPublic  | BindingFlags.Instance  | BindingFlags.GetField,  null,  runtime,  null);

ApplicationShutdownReason shutdownReason = System.Web.Hosting.HostingEnvironment.ShutdownReason;

NLog.Logger logger = NLog.LogManager.GetCurrentClassLogger();
logger.Debug(String.Format("\r\n\r\nAPPLICATION END\r\n\r\n_shutDownReason = {2}\r\n\r\n _shutDownMessage = {0}\r\n\r\n_shutDownStack = {1}\r\n\r\n",
                shutDownMessage, shutDownStack, shutdownReason));
}

void Application_Error(object sender, EventArgs e) 
{
    NLog.Logger logger = NLog.LogManager.GetCurrentClassLogger();
    logger.Debug("\r\n\r\nApplication_Error\r\n\r\n");

}

Our log file is littered with "APPLICATION STARTING" entries, but neither Application_OnEnd, Application_End, nor Application_Error are ever fired during these spontaneous restarts. I know they are working because there are entries for touching the web.config or /bin files. We also ran a memory overload test and can trigger an OutOfMemoryException which is caught in Application_Error.

We are trying to determine whether the virtual memory limit is causing the recycling. We have added GC.GetTotalMemory(false) throughout the code, but this is for all of .Net, not just our App´s pool, correct? We've also tried

var oPerfCounter = new PerformanceCounter();
oPerfCounter.CategoryName = "Process";
oPerfCounter.CounterName = "Virtual Bytes";
oPerfCounter.InstanceName = "iisExpress";
logger.Debug("Virtual Bytes: " + oPerfCounter.RawValue + " bytes");

but don't have permission in shared hosting.

I've monitored the app on a dev server with the same requests that caused the recycles in production with ANTS Memory Profiler attached and can't seem to find a culprit. We have also run it with a debugger attached in dev to check for uncaught exceptions in spawned threads that might cause the app to abort.

My questions are these:

How can I effectively monitor memory usage in shared hosting to tell how much my application is consuming prior to an application recycle?
Why are the Application_[End/OnEnd/Error] handlers in global.asax not being called?
How else can I determine what is causing these recycles?

Thanks.

EDIT: Based on answer by @Jani Hyytiäinen

Scenario: Thread #1 begins and is followed by thread #2. Thread #1 hits memory limit but continues processing. Thread #3 begins. Thread #1 finishes, but #2 processes more than 60 seconds after #1 has hit the memory limit.

The pool then ungracefully aborts? What http responses will #2 & #3 receive (These are AJAX calls, but I am getting 504 errors in Fiddler)?
Is the request for #3 accepted or does it just queue until a new pool has started up?
Is there any way to know that the memory limit has been hit or is about to be?

Any strategies welcome.

The HTTP 504 Gateway Timeout not actually returned by your site since the error itself means the resource requested by url did not respond in a timely manner. In your case, probably because the app pool was shut down and new one hadn't yet started up. When application pool shuts down, all the threads in it will abort with it. Which most definitely leads to HTTP 504 on threads #2 and #3 — Jani Hyytiäinen, Dec 13 '13 at 20:37

score 16 · Accepted Answer · answered Nov 14 '12 at 23:04

There is an application pool shutdown time limit in iis. Let's say it's 60 seconds and in shared hosting environment there is an application pool memory limit. Your application pool reaches this limit and iis tells the application pool to finish all the work with current requests. If all the requests finish processing before 60 seconds is passed, application_end will trigger and application pool will gracefully shut itself down. However if 60 seconds pass and requests are still being processed, IIS gets upset and kills the app pool. This time no application_end would trigger. Similarly, no error event handler would be triggered.

So, is there no reliable way to do something when an ASP.NET application ends? — Water Cooler v2, Apr 23 '20 at 06:30

score 0 · Answer 2 · edited May 23 '17 at 12:02

0

I took a look at one of my applications that fires shutdown events correctly, and here is the signature in Global.asax.cs:

protected void Application_End(object sender, EventArgs e)
{
    ApplicationShutdownReason shutdownReason = System.Web.Hosting.HostingEnvironment.ShutdownReason;
    // Write to log
}

The only difference I notice is that your "Application_End" and "Application_Error" methods do not have the "protected" modifier. I've seen plenty of situations where reflection doesn't work with private members, so that might be at play.

Also, try setting your idle timeout on your local/dev application pool to something low (like 1 minute), and then make sure your Application_End event fires locally.

Note that by default IIS application pools recycle after 20 minutes of no requests, so you might be hitting that if you have a low-traffic application. Also, some shared hosting providers change this "idle timeout" value to be lower to conserve resources.

edited May 23 '17 at 12:02

Community

1
1

answered Nov 14 '12 at 22:57

Vlad Magdalin

1,692
14
17

1

Thanks, but like I said we're getting Application_End event for other types of shutdown, "I know they are working because there are entries for touching the web.config or /bin files." Cute pic btw:) – Laramie Nov 14 '12 at 23:32
1

@Laramie, sorry, I misunderstood. Have you tried to set your dev environment [virtual memory limit](http://technet.microsoft.com/en-us/library/cc753179(v=ws.10).aspx) to something really low to see if you could reproduce the behavior you're seeing in the shared environment? – Vlad Magdalin Nov 15 '12 at 01:49
We're on to that next so we can track memory usage. In the meantime we've been optimizing and have it under control for the time being. I don't think there will be an easy answer here, but I'd be thrilled to upvote one! – Laramie Nov 15 '12 at 06:30

score 0 · Answer 3 · answered May 28 '13 at 07:59

Not sure what version of IIS you're using but on IIS7.5 there's a bunch of App Pool Recycle Event Log settings that allow you to log the reasons for app pool recycling.

In IIS go to Application Pools and go to Advanced Settings on the app pool in question. Go down to Recycling and there's and sub-menu called "Generate Recycle Event Log Entry".

Should Application_End fire on an automatic App Pool Recycle?

3 Answers3

Linked