Azure Web App (ASP.NET MVC) becomes cold every ten minutes and takes +10-20s to load

Question

I have a very weird issue with an Azure Web App, and I'm getting quite frustrated with it.

We experience our app is very fast and responsive when using it, however, if we don't use it for roughly ten minutes, it has a very cold start (~10-20 seconds). This cold start only happens when it involves the database. When it's a bit like when we release the web app.

Our attempts

Using Application Insights inside Azure, we have setup this ping every 5 minutes:

The outliers are always caused by my deploys (not using deployment slots right now). However, this login page does not call our database so we don't see the "cold" start in these data.

The application setup should be solid. Our web app is hosted in North Europe with Always on:

We just moved the whole setup to a new resource group / app service plan, to make sure our problem was tangled with our other apps. The new app service plan is a Standard 1 small, which should not be a problem. Looking at our consumption I am not worried, and could probably even try a smaller service which I will do after solving our problem:

Our SQL database is also hosted in North Europe (checked locations a billion times because I've made that mistake before).

Just like with the app service, we've picked a "too big" hardware to make sure that is not causing the problem (the Standard S0: 10 DTUs). The usage is ridiculously low:

We do use continuous deployment (Deployment options inside the Azure menu), but looking at the deployments, it should not constantly deploy something:

The frustration comes in the app is super responsive when it works. When it's "warm" every page loads in seconds, just like my average response time shows on our web app:

But these numbers are just plain wrong when we (or our users!) use our app. Here we experience it's very often a +10-20 second load the first time.

Does anyone have ANY idea? Any hints? You've no idea how grateful I will be.

EDIT & UPDATE 1:

I have decided to setup some more tests. I've now managed to get the real data showing our problem by calling another page. Ironically this page do NOT call the database, so while I thought this was a database problem, it does not seem like this. See the challenge here (trend continues +24 hours).

It's weird how stable it is being exactly ~10 seconds. And the trend does not seem to be every 10-20 minute, but closer to every 5 minute - with exactly the same interval between them:

EDIT & UPDATE 2:

I've been digging some more. Turns out there are a couple of very interesting insights: The "slow" 11 seconds calls from edit 1, is only from East US and from one endpoint (http://prntscr.com/jcv69w), and

The most interesting thing I found is the following:

The application itself does NOT have any caching. I use Entity Framework which I assume use some caching, but that's all.

I was logged into our app, and clicked around in Chrome. I found out, that the pages I had already visited was showing instant (with data from DB), but if I opened a new page, it would load slow. It seemed like some entities is being cached the first time I open a page.

I then tried to open the app in a new browser. If I opened a page I had prior opened in Chrome, it would open instant. If I opened a new page I didn't click before, it would have the ~10 second load.

My best guess right now is that the Entity Framework I use is giving problems for some reason.

EDIT 3:

Just added a bounty, and is setting up a lot of logging. I have added the MiniProfiler, but have trouble getting it to work in production (is only shown on local requests).

I have also added logging in global.asax for Application_Start and Application_BeginRequest and Application_EndRequest to see some and status there. Will update with findings soon.

EDIT 4:

So now I've the first interesting numbers in. The app is not being recycled. Application_Start is only called once.

I can see the time difference by logging on EndRequest and BeginRequest. I can see there are multiple calls which takes more than +15 seconds between these two... But when site is warm, it takes ~0.5-2 seconds depending on page. So something very weird is happening between the beginning and end of the request. Debugging further!

EDIT 5:

Got MiniProfiler to work. Here is an example of the slow load (~15 seconds):

My next step is adding Entity Framework tracking and even some more line for line calls. I'm getting my money on the database!

EDIT 6:

Okidoki, I was wrong. it's the render method that's slow - not the database! I've NO idea how to debug this... To the google!

EDIT 7:

Time for another update. Status is: nothing has been solved.

So I have tried a lot of things:

1) I tried to disable all types of caching (Prevent Caching in ASP.NET MVC for specific actions using an attribute) and I have same behavior. First load? Slow. Next load? Fast. Wait 5-10 minute, same behavior so not solved.

2) I had some custom things in my startup.auth file with a 5 minute delay. Removed. Not the problem.

3) I used a custom attribute for authorization. I removed that.

4) I updated my Entity Framework implementation to make it work in per request

I'm getting really frustrated. My next step is:

A) Try to make 5-10 versions of same page (without _layout, with layout, with database, without database, with dependency injection, without... all these things), so see if I can find a pattern.

B) Try moving the hosting to a virtual machine to see if it solves the problem

EDIT 8 - NEW RELIC ADDED:

I have now added New Relic. Two very scary things are the following (I found and reproduced the error!):

And frontend wise (Browser part of New Relic), there is a ~15s lack between two starts:

http://prntscr.com/jevgeg vs http://prntscr.com/jevgix with nothing inbetween.

Answer 1

I have a couple of possible answers.

Entity framework code-first/database initialization: If you are using code first setup with migrations and possibly seed data, each of these things can cause some "warmup" issues.

Specially if you are not initializing the database on app startup, that would mean that the first time you hit your database is when it gets initialized.

Entity framwork version: Entity framework itself has also had a lot of performance improvements in 5 and 6.x some of these also have to do with both cold and warm startup speeds.

Views aren't precompiled: If you are getting slow loads on pages (like after deployment) every first hit of a new page (view), and then subsequent loads are fine. This can be because pages aren't compiled, i can elaborate on that if thats the case.

Recycle It sounds like you are experiencing these issues when the application recycles, and it isn't auto initializing (which is why you get that cold hit) the worst performance issues i have seen with these things are usually entity framework and precompile related. But both can be easily fixed. But ensuring the app is "always running" and self initializes after recycle also ensures that no users get this cold hit.

UPDATE: Since it was view related i can offer a solution i have found very useful. Installing RazorGenerator.Mvc Nuget package. And adding this Engine as the first engine will ensure you use compiled views.

In App_Start you could create a file called RazorGeneratorMvcStart.cs with content like this:

using RazorGenerator.Mvc;

[assembly: WebActivatorEx.PostApplicationStartMethod(typeof(MyNamespace.RazorGeneratorMvcStart), "Start")]

namespace MyNamespace {
    public static class RazorGeneratorMvcStart {
        public static void Start() {
            ViewEngines.Engines.Insert(0, new PrecompiledMvcEngine(typeof(RazorGeneratorMvcStart).Assembly));

            VirtualPathFactoryManager.RegisterVirtualPathFactory(engine);
        }
    }
}

The razor engine can even take a parameter for UsePhysicalViewsIfNewer for those that like replacing a view live. In that case it uses the precompiled version, unless a view with a newer date than the compiled .dll has been placed in the folder.

This approach should solve performance issues with views.

Answer 2

A few ideas:

1. On your Web App blade, go to the Diagnose and solve problems menu. Then click on performance counters. I'd honestly fish through every one of the available perf counters, paying attention to the timeline vs your degraded performance. I once found out that SignalR was choking out my server due to runaway connections by looking at Thread Count.

2. Is the server errors log in Application Insights clean?

3. Under the Diagnose and Solve Problems screen, do you see anything suspicious in the Failed Request Tracing Logs?

Answer 3

1. Separate out anything from deployment vs local. If the application is running perfect in local environment, then something different is happening when it goes to Azure. Resolving something is taking lot of time.
2. Any static resource (script, styles etc) do get auto cached by browser on 1st request, so on subsequent request, the issue should not come.
3. Since you know that "Render" method is giving the problem, looks like a complex nesting computation and browser DOM manipulation is happening. However if this is not a challenge when on local, then check if rendering calls for outside resources and they getting blocking calls (may need heavy use of ajax or server side async calls, although I assume you would already have that in place.)
4. Possible refactoring may be required (to have smaller stacks and possible use of threading/non-blocking IO), however require to see your code to suggest on that.

Share your page load event and any subsequent calls made by it. Also how's the view is rendered along with any DOM handling.

Answer 4

There are two more options to find and fix.

1. User New Relic to trace.

Check this post from hanselman on using it in Azure.

2. Follow Asp.Net MVC Best practices

Check this post for Best practices.

Also for data reads you can Use ADO.NET with stored procedures instead of Entity Framework, because as of now EF may have some performance issues.

Answer 5

I am posting an answer even though it's not solved, but I'm 99% sure I've found the underlying problem.

What happens is that when I release, every view needs to be built. It takes ~15 seconds to build a view, which is what New Relic also shows in my latest update.

This brings two temporary solutions and one bigger question: why is the building of the view so slow?

The temporary solutions are simple. Either compile the views when releasing or visit the most important pages just after a release. This is obviously annoying because I release multiple times per day.

The reason it's so slow, I assume, is because I am using a very big Bootstrap theme. The way I am handling the bundles is not very efficient, which could provide a problem.

The reason I thought the pain was the site was slow after ~10 minutes, was simply because I released new code quite often, and not visiting a large portion of our pages. After doing this, it's quick.

Thanks a lot for your help and support - at least now I can deal with it.

Answer 6

Are you compiling Less or Sass in your bundling setup?
If yes, what JavaScriptEngineSwitcher are you using?
I remember having a similar problem. The bundles are compiled on first access and it was taking a huge amount of time.
The solution was to switch to the V8 engine.