15

We are having issues with an Azure Application Service. One of our webservices (MVC) caches data from the database at startup (Application_Start) - this takes approximately 3 minutes. Until this is ready we can't handle requests.

This is known so we set it 'always on' and will aim to only restart it during off-peak times if necessary.

However, we expect heavy load to the server next month, and in our testing of the auto-scaling, we have found that when it adds additional instances, each of these instances goes through the same startup delay - but the traffic is split between the current running instance and the new one that's warming up so e.g. half of the requests start failing for that 3 minute period.

How can we configure Azure to delay using the new instance until it is ready? (or should we be using e.g. AWS instead?).

Some of the documentation points to using a custom Load Balancer Probe however it mainly talks about VM's whereas we are using PAAS.

Lance
  • 312
  • 4
  • 11
  • Can you do something like a 302 redirect to the main instance until the caching is complete? – Brendan Green Oct 31 '17 at 00:27
  • 2
    they all have the same url - the additional instances are invisible to the end user – Lance Oct 31 '17 at 00:34
  • Of course. The only other thing I can think of is to use traffic manager in front of the websites, and enable/disable endpoints as they are coming online. But that of course poses a problem with multiple instances that sit behind a single app service instance. Which then poses the other questions - what is being cached? If it is **not** unique to the instance, can this be moved into something like redis instead? That way you have a single cache instance of the data, and websites will become immediately available on a scale event – Brendan Green Oct 31 '17 at 01:16
  • I spent a lot of time trying to get it to work with REDIS but the data isn't suited to key/value matching (as I'm doing a fuzzy match - levenschtein distance over a large number of keys), so in RAM directly works so much faster which is important – Lance Oct 31 '17 at 21:23
  • 1
    I've also tried using the initialisation in web.config, where my warmup doesn't return until the data is loaded, but it's not firing at all – Lance Nov 02 '17 at 21:02
  • @BrendanGreen this is what I was trying to achieve in [this question](https://stackoverflow.com/questions/47437243/returning-a-custom-response-code-for-for-serving-static-html-when-using-iis-appl). Any ideas how? – Frank Fu Feb 26 '18 at 02:42
  • Some thoughts on this topic are at https://www.jan-v.nl/post/warming-up-your-app-service. Maybe this helps? – Henry Been Apr 24 '19 at 06:18

4 Answers4

1

Do try to reduce the data you need to load on app_start and try to lazy load data into Cache on first request. Some times even after doing all of this we do end up with large sets of data that is necessary on start.

There are two ways we can approach this.

One, assuming you are using in-memory caching and every instance of the app needs to hydrate its in-memory cache on App_Start. Try to use a external cache provider like Azure Cache for Redis, your new instance can just point to this external cache without having to reload the data.

Two, you can depend on Application Initialization Module which was introduced in IIS 7.5 (installed on Azure App Services' IIS). To use this feature, you need to add applicationInitialization section under web.server section of web.config. This will help you not make the instance available until the warm-up process is completed. More info on how to use ApplicationInitialization is available in this blog post

The best case would be to use the combination of both, applicationInitialization will point at a page in your application which checks if the external cache is available and hydrated, if yes, complete, else hydrate the external cache.

Sai Puli
  • 951
  • 8
  • 12
0

You can do this in Azure with other resource type than classic VM like an App Service. App Services scale up and down with instances that share the same memory pool and thread pool.

Simon Bourdeau
  • 429
  • 5
  • 18
0

There is a lot of good information, in the link https://www.jan-v.nl/post/warming-up-your-app-service that was included in one of the comments.

Based on that information the functionality that you require is not available in the free tier.

I would approach the problem differently. Why does it take 3 mins to load the data from the database? Since it is only loaded on start it should be data that does not change often.

Could you:

  • Optimise the reading of data from the database?
  • Reduce the amount of data you read from the database?
  • Export the data to a file, and read it from a file?
Shiraz Bhaiji
  • 64,065
  • 34
  • 143
  • 252
-1

My recommendation would be to use an Azure Load Balancer with a health probe

Cortese1127
  • 1
  • 1