Azure web app worker process recycling unexpectedly

Question

We have an ASP.NET MVC web site running in an Azure Standard Medium App Service plan.

Occasionally the worker process is recycled without warning.

I added code to Application_End in global.asax.cs to log the HostingEnvironment.ShutdownReason.

The logged reason is HostingEnvironment which as I understand it means the hosting environment initiated the shutdown.

There are no application errors or events logged. "Always On" in Application Settings is set to "On".

How can I determine the root cause of the shutdown?

**FYI:** Application_End will not fire if process doesn't exist gracefully. In the words, if application crash, it will not fire. — Win, May 05 '16 at 00:11

score 1 · Answer 1 · answered May 05 '16 at 02:36

1

I'm not too familiar with Azure so perhaps this isn't relevant but by default IIS recycles app pools every 29 hours unless you specify otherwise.

some more info here.

answered May 05 '16 at 02:36

Christopher Thomas

382
1
15

1

According to the accepted answer at http://stackoverflow.com/questions/25545677/meaning-of-always-on-setting-on-azure-web-site Always On disables periodic recycles. This jives with the fact that I've had other app pools stay up over 29 hours in Azure. – Kevin Krueger May 06 '16 at 23:16

score 0 · Answer 2 · answered May 05 '16 at 21:51

Any chance you've found any 500.121 errors in your web logs? I experienced an issue with a deadlock inside of the w3wp process that resulted in zero I/O for awhile. It turns out azure slams the door on any processes with zero I/O for about 4 minutes.

The entries in the log will jump out at you. Most of the W3C properties normally listed were empty (including the client IP!)

Ended up being a code bug.

score 0 · Accepted Answer · edited May 23 '17 at 10:30

After adding more detailed shutdown reason logging as described here, I found the following details logged:

"Shutdown message: Change Notification for critical directories.
File Change Notification Error in App_LocalResources
HostingEnvironment initiated shutdown
Change Notification for critical directories.
File Change Notification Error in App_LocalResources
Change Notification for critical directories.
File Change Notification Error in App_LocalResources
Change Notification for critical directories.
File Change Notification Error in App_LocalResources
Change Notification for critical directories.
File Change Notification Error in App_LocalResources
Change Notification for critical directories.
File Change Notification Error in App_LocalResources
Change Notification for critical directories.
File Change Notification Error in App_LocalResources
Change Notification for critical directories.
File Change Notification Error in App_LocalResources
CONFIG change
CONFIG change

This leads me to believe that there are some transient issues happening in Azure causing ASP.NET change notification to fail, thus triggering the application to restart. A similar issue is described in this question.

As a work-around I have added fcnMode = "disabled" to the httpRuntime element in the web.config. The root cause remains unknown. The accepted answer in the linked question indicates the cause in that case was an SMB bug in the Northern Europe data center. There must be other causes, because I experienced this issue in the US East datacenter almost 3 years later.

Azure web app worker process recycling unexpectedly

3 Answers3