IIS Application Initialization in Azure shows 500's periodically after an Upgrade Deployment

Question

Recently, I implemented Application Initialization in one of our Azure projects in order to minimise the time it takes for each of the sites to warm up.

I essentially have implemented near exactly the steps outlined in This blog post.

On a fresh deployment, it works for all 10 of my sites in IIS. The problem is though, when I then try and Upgrade that deployment I get 500 errors for the first 25 requests or so for each site.

Now, this isn't ideal, as instead of hitting each site once to warmup after a deployment, I now have to load each site around 25 times before I get past the 500s. They don't stop there though, they seem to sporadically happen.

If I re-image the machines post-deployment, that fixes the issue. At the cost of downtime though, which isn't an option.

Any ideas?

I have the following:

Startup task in ServiceDefinition.csdef to turn on the Application Initialization Module:

<Task commandLine="enableApplicationInitializationIIS.cmd" executionContext="elevated"></Task>

Then inside that task I have:

PKGMGR.EXE /iu:IIS-ApplicationInit

In the ServiceConfiguration.csfg I have set up the latest osFamily:

<ServiceConfiguration serviceName="Foo" xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceConfiguration" osFamily="3" osVersion="*" schemaVersion="2012-10.1.8">

Then in the Web Role, I have the following to turn on all the Application Initialization required settings:

public class WebRole : RoleEntryPoint
{
    public override bool OnStart()
    {
        using (var serverManager = new ServerManager())
        {
            foreach (var site in serverManager.Sites)
            {
                foreach (var application in site.Applications)
                {
                    application["preloadEnabled"] = true;
                }

                site.ServerAutoStart = true;
            }

            serverManager.ApplicationPoolDefaults.ProcessModel.IdleTimeout = new TimeSpan(00, 00, 00);
            serverManager.ApplicationPoolDefaults.Recycling.PeriodicRestart.Time = new TimeSpan(00, 00, 00);
            serverManager.ApplicationPoolDefaults["startMode"] = "AlwaysRunning";

            foreach (var appPool in serverManager.ApplicationPools)
            {
                appPool["startMode"] = "AlwaysRunning";
            }

            serverManager.CommitChanges();
        }

        return true;
    }
}

And finally, each of my sites have the following:

<system.webServer>
    <applicationInitialization skipManagedModules="true">
        <add initializationPage="/" />
    </applicationInitialization>
</system.webServer>

Not sure what skipManagedModules is though?

@SandrinoDiMattia As it's your blog post, I was wondering if you could provide any insights? Thanks. — Mathew Thompson, May 17 '13 at 07:51
Well, the only thing I have to say is that in-place upgrades are just a bad idea. What if the upgrade fails in the middle? You should create a staging deployment, then swap it. — sharptooth, May 17 '13 at 08:16
@sharptooth We do have 2 deployments, staging and production. I'm testing this thoroughly on staging before it goes to production. Then we'll deploy production to staging, then do the swap. — Mathew Thompson, May 17 '13 at 08:24
@sharptooth We do because we Upgrade Staging, then do the switch to production. Which would yield the 500s — Mathew Thompson, May 17 '13 at 08:31
This is Very Bad Idea, because if if slightly trash your staging that damage will persist forever and it will not be reproducible. The right way is that you create a new staging, check it, swap, then delete the previous production (which is now staging). — sharptooth, May 17 '13 at 08:42
@sharptooth Fair point, but that's not really an option in our scenario as we'd have to update all of the DNS entries for each deployment. — Mathew Thompson, May 17 '13 at 09:03
Why? Just have a CNAME mapping to yourservice.cloudapp.net and then your URL will always be mapped to production deployment. — sharptooth, May 17 '13 at 09:07
@sharptooth The Staging instance doesn't have CNAME mapping, they're only for Production. Staging gets a random string of characters for it's CNAME. — Mathew Thompson, May 17 '13 at 09:20
Do you have the event logs on the machine to see what the cause of the 500 errors are ? Have you RDP in to have a look? Do you connect to for example a database some cache that might be having problems in the upgrade situation ? For example Cache / Database might be running out of connections? — JamesKn, May 17 '13 at 11:10
You are getting 500s because you have clients hitting the server before it is fully deployed. We constantly update a production site with 10,000 users per day. We follow the standard Azure procedures - deploy new code to staging and test @ .cloudapp.net, then when ready, swap the VIP. The Azure load balancer gracefully handles moving clients from old instances to new instances. — viperguynaz, May 18 '13 at 04:17

score 2 · Answer 1 · edited May 23 '17 at 11:51

I experience a similar issue and found your post. In my case this issue was already there with an initial deployment and as well with an upgrade deployment.

In my error logs I found a "Value cannot be null" exception. It seems that the ServerManager was not able to initialize - the code inside the using statement was not executed. It turned out I tried to deploy the assembly Microsoft.Web.Administration (from Windows 7) to a Windows Server 2012 machine.

After applying the correct version, the 500 errors disappeared.

System.TypeInitializationException: The type initializer for 'Microsoft.Web.Administration.ConfigurationManager' threw an exception. 
System.ArgumentNullException: Value cannot be null.
at System.Activator.CreateInstance(Type type, Boolean nonPublic)
at System.Activator.CreateInstance(Type type)
at Microsoft.Web.Administration.ConfigurationManager..cctor()
--- End of inner exception stack trace ---
at Microsoft.Web.Administration.ConfigurationManager..ctor(ServerManager owner, String applicationHostConfigurationPath)
at Microsoft.Web.Administration.ServerManager..ctor(String applicationHostConfigurationPath)

EDIT:

I continued to experience 500 Errors at initialization and it seems that these happen because of 2 tags:

<applicationInitialization remapManagedRequestsTo="app_starting.htm" skipManagedModules="true" >
    <add initializationPage="/" />
</applicationInitialization>

Removing the remapManagedRequestsTo and skipManagedModules attributes seems to remove the 500 errors see IIS 7.5 Application Initialization for ASP.NET web service (warmup) without remapping requests.

IIS Application Initialization in Azure shows 500's periodically after an Upgrade Deployment

1 Answers1