39

I've been investigating a slow warmup time for my production app and I got some unexpected results.

Testing on various Azure VMs showed the warmup time to be well-related to the application server performance (so it's not external request, including sql), but what's really interesting is the comparison to Azure Web Apps and my own local machine (all times are averages from at about 3 runs):

VM A0:            >1m
VM A2:            13s
VM D2V2:           6.8s
VM D5V2:           7.8s
Wep App P2:       25s
Web App S2:       26.5s
My local machine:  6.6s

My local machine is an i5 with 3.3GHz.

The local machine is fastest even though it connects to the same SQL Azure database over the internet and the request includes Entity Frameworks's model checking.

All tests use .NET 4.6.1, the newest release at this time.

Apparent conclusion:

  • My 4 year old desktop box is faster than any size of Azure VM for jobs that are not parallelizable.
  • Azure Web Apps run on potatoes even if you're willing to spend 500 bucks a month.

This seems fishy. Any ideas what else might be going on? Or what to test?

EDIT after some profiling: None of the following sheds any light on my question, but it's still interesting information (tested on another A2 VM with 20s total request time while the profiler being attached):

  • 58% is jitting.
  • File IO blocking is virtually zero (the app has been started before, so I guess all required dll parts are in the memory cache).
  • <1% is SQL Azure requests.
  • The rest of the 90% running time should therefore be .NET execution or profiler overhead.
  • The profiler itself has quite the overhead, as the request runs in about only 15s without it (I used a trial of JetBrains dotTrace, which I really like)
  • 50% of the time is during the first request with Entity Framework, 80% of it being jitting
  • 15% being the first use of SignalR, almost no jitting (doing some reflection-nonsense)

Almost no parallelization is being done.

John
  • 6,693
  • 3
  • 51
  • 90
  • why dont you try publishing it as a cloud service instead.Im a newbie to azure too and when i hosted my website as a azure website it was kinda slow but as a cloudapp it gave a good response time – Sujit.Warrier May 01 '16 at 13:46
  • Is this a Web App or are you hosting this a dedicated VM? – viperguynaz May 01 '16 at 15:30
  • @viperguynaz Web App, but I tested VMs too, see the timing table. – John May 01 '16 at 15:54
  • What disk drive were you using in your azure VM tests? Can you describe the workload done during "Warm-up"? How fast were subsequent requests? – CtrlDot May 01 '16 at 16:59
  • @CtrlDot Do I have a say in what drives are used? I thought that's normal blob storage. I've added some profiler results. Subsequent requests are not particularly slow. – John May 01 '16 at 17:14
  • In Azure VMs, c: drives are blob, but the D: drive is SSD local to the box. Further, if you add additional drives, there can be rate/throughput limits depending on configuration – CtrlDot May 02 '16 at 00:40
  • @CtrlDot The tests were on C:, the blob storage. However, the tests were also warm starts almost not blocking on file io at all, so it shouldn't matter. – John May 02 '16 at 08:13
  • 4
    Not helpful to the OP, but I found myself here searching for why my azure webapp seemed so slow, and it transpired my web-app was being hosted in a different location to my database/caching/blobs/etc... – James Jul 14 '16 at 13:57
  • @john did you every come to any sort of resolution to your issue? I'm running a node.js app and am seeing similar results. I am running a S2 Standard instance. I have an endpoint that's taking nearly 30 seconds to returns and while I know the db function behind this endpoint is slow, it's not that slow. I profiled it on my own machine hitting the same production database and I'm coming in under 10 seconds (not fast, but not 30 seconds!). I'm tempted to fire up a VM on another service for comparison. – jdixon04 Feb 08 '17 at 20:54
  • @jdixon04 Is the db executing on the web app machine itself? If so, it would probably be sensible to move the db to a different machine or use SQL Azure as I do. If it's not on the same machine, the speed of the web app machine shouldn't matter in your scenario. I would have thought that the slowness is somewhat fine with node as node, unlike .NET, doesn't have an expensive warmup (or does it?). – John Feb 08 '17 at 21:02
  • @John the database is on a separate Azure VM located in the same region as my Node.js app running on Azure App Service. My concern is that the Azure App Service itself seems to be performing very poorly in comparison to my local machine, running the same app, hitting the same database. Even if I bump the instance to S3 Standard (4 cores, 7gb RAM), I'm not seeing any significant gains. – jdixon04 Feb 08 '17 at 21:09
  • @jdixon04 But what do you do that's CPU bound on the app service? Normally reasonable web traffic doesn't cause a high CPU load, which is probably why MS thought they could get away with such slow machines. My problem was also only the warmup, which is computationally expensive in .NET, in part due to just-in-time compilation. – John Feb 08 '17 at 21:12
  • @john for the case i mentioned above, not a lot at all. The expensive computation happens in the database, which my applications calls and then returns the data. The other thing I'm seeing is that it doesn't seem to be handling parallel requests well. For example, the endpoint I'm talking about returns appointments to a calendar. On the frontend, that endpoint is called, but other endpoints are called lazily to load associated appointment data. On my local machine, those other requests quickly process, but on Azure, they seem to be bound by the call to the first endpoint. – jdixon04 Feb 08 '17 at 21:16
  • I'm wondering it that parallelization issue has something to do with IISNode, which Azure uses to run Node applications. – jdixon04 Feb 08 '17 at 21:17
  • @jdixon04 I've not used node yet, but I thought a node app is always entirely asynchronous. If that's the case, the app service should handle a huge number of requests simultaneously even if the db is slow to answer. The way you describe the setup I don't think that a faster app service would make the responses come faster. Not sure how to debug the problem though. – John Feb 08 '17 at 21:20
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/135217/discussion-between-jdixon04-and-john). – jdixon04 Feb 08 '17 at 21:21
  • I also can't understand any real reason to use Azure. It's slow, VERY expensive, difficult to configure and without any real technical support who could have access to machines. All the benefits they promise in my opinion not really true. In my experience maintaining dedicated machines takes much less time, cost much cheaper and works really great. – Andrey Feb 20 '17 at 13:55
  • @AndreySapunov I agree for simple web space, but many of their services, such as SQL Azure or blob storage, are redudant or even distributed. It's certainly not cheaper to re-implement those services, and run them on your own hardware with regular backups, software updates, etc. Also, you need to buy their support. Once you have a support plan, the support is fine. – John Feb 20 '17 at 14:03
  • @John What did you end up doing? I'm getting fed up with a one minute warm up time with my web app and really slow CPU. It looks web apps still use old A series VMs, instead of d series which are 70% faster. My thought is I will need to move to hosting the web app in iis in a dedicated VM instead of using app services. – TWilly Aug 06 '17 at 18:06
  • @TWilly A minute is still a bit more than the worst I had so far, but I had to do this (migrating to VMs) twice also for other reasons: Once a bug in .NET that wasn't fixed on app services and once some socket starvation issue I can't get to the bottom of. App services just lack the final bit of maturity that they really should have I think - they are still very useful to get a new app up quickly. – John Aug 06 '17 at 20:19
  • https://azure.microsoft.com/en-us/blog/announcing-general-availability-of-the-new-app-service-premium-plan/ – Dave New Oct 02 '17 at 13:30
  • 1
    The new Dv2 machines are much faster, but are only support on the over-expensive Premium tier. See feature request https://feedback.azure.com/forums/169385-web-apps/suggestions/33046582-support-dv2-series-vms-in-standard-tier – Dave New Jan 23 '18 at 11:40
  • @davenewza Back when I posted this I tried premium and it was slow too (see the results). So I take it that they use Dv2 machines now is new? I would consider it then even for the high price. – John Jan 23 '18 at 12:07
  • 1
    @John Yes, Premium v2 went into general availability end of 2017. We are experiencing just over 2x performance improvements on our CPU-intensive workloads. A-series to Dv2 is a big jump. – Dave New Jan 23 '18 at 12:18
  • @davenewza I just came round to test this, and I still can't measure any substantial speed difference between premium and standard. Is there something special I should do? – John Mar 01 '18 at 20:10
  • 1
    @John are you sure you have configured it as a Dv2 instance? The premium tier offers both. – Dave New Mar 01 '18 at 20:19
  • 1
    @davenewza Got it now, they are indeed twice as fast. Thanks. – John Mar 02 '18 at 10:27
  • @davenewza Any ideas why premium is 4x as expensive as standard? I can't get over the fact that we need to pay 4x for a faster cpu and ssd, sigh – TWilly Apr 05 '18 at 21:35
  • @TWilly: Yes, because it's on the Premium tier (and not on Standard). Please comment & upvote here to request that they offer Dv2 on the Standard tier - https://feedback.azure.com/forums/169385-web-apps/suggestions/33046582-support-dv2-series-vms-in-standard-tier – Dave New Apr 06 '18 at 08:36
  • Facing the same problem with slow startup. May I ask how you profiled the startup time? – viblo Oct 22 '19 at 11:44
  • @viblo dottrace by Jetbrains, a really neat product. I use it only ever in trial because I can't buy it without buying the whole of resharper, which I have no use for. – John Oct 23 '19 at 08:49

2 Answers2

5

Initially when we deployed our Azure App Service - Web app (hosted in Resource Group1) and our Azure SQL database (hosted in Resource Group2), the app was very slow. The mistake that I made was that both the RG1 vand RG2 were in two different locations. Later, when we changed the location of both RGs (recreated them actually) to be the same then the WebApps started running smooth.

Thanks, Prawin

Prawin
  • 1,158
  • 2
  • 12
  • 26
1

I too have the same problem, I fixed by moving or copying my site contents(Pictures and Videos) to Azure Blob Storage. We can also configure Azure CDN to WebApps, VM, Cloud Services and Custom origin. By trying thsi Azure CDN your site loading time will improve.

Rajasekar
  • 129
  • 2
  • 12