1

We have a product made of two components. A WEB API site hosted under IIS 8 which I call it API. This site provides services to multiple sites however at the moment only one web site is using it. The other component of this site is a ASP.NET MVC (.NET 4) site that is at the moment the only client of the API site. Let's call it WEB.

When we deployed the API and WEB to a single server with production capacity everything was fine until when the number of concurrent users exceeded 16! Then the site became unresponsive which means that we did not get a Timeout (or any other) errors and the browser was waiting for a response for good.

We reproduced the same issue in our staging environment using an online load test web site. Again the site becomes unresponsive when the number of concurrent users exceeds 12 or 16 (or higher).

When this happens the site kind of dies for like 10 minutes and then becomes responsive. I guess this is because the App Poll gets recycled?

The question is that how can I know if the problem resides in the API or in WEB? Then what could be the possible issue? WEB totally relies on API and does not connect to any databases directly but API is using MySql on AWS.

JonK
  • 2,097
  • 2
  • 25
  • 36
Aref Karimi
  • 1,822
  • 4
  • 27
  • 46

3 Answers3

1

As some users pointed out here, the problem was thread exhaustion (aka thread contention). We tried to identify and amend any code that could block the threads. We also made loads of performance improvements to the API and site level to help the threads finish quickly. Finally we deployed the site to the production environment and got an acceptable performance.

Aref Karimi
  • 1,822
  • 4
  • 27
  • 46
0

Have you looked at what is eating up all your resources? Is the memory maxed out? Processor maxed out? Where is your bottle neck? 16 users should not eat up all the threads in an app pool by any means. I would guess some process is eating up all your memory on your server rendering it unresponsive. Find out what that process is with Process Explorer or any program that monitors your servers resources.

Jason Roell
  • 6,679
  • 4
  • 21
  • 28
  • Thanks Jason, On the sever CPU rate goes up to 50% and then drops to 5% when the site crashes! Memory usage never exceeds 3GB (of 15GB). I ran LeanSentry diagnosis tool which says the issue is at "global.asax/ExecuteRequestHandler". Can it be that API does not respond in a timely manner so that the app pool of WEB gets stuffed by new incoming requests? – Aref Karimi Jan 27 '14 at 02:24
0

Have you tried adding more worker processes in IIS to the appools? We had some perf issues with an MVC site that we fixed by simply adding more worker processes.

spaceman
  • 1,628
  • 1
  • 16
  • 19
  • How would I add more worker processes? Is it equal to increasing the maximum number of threads in the app poo? And is not this like just putting band-aid on the problem? I read on Stackoverflow that using AsyncController might help, is that true? – Aref Karimi Jan 27 '14 at 22:45
  • I installed LeanSentry monitoring tool on the server and from its reports I can tell that the IIS threadpool gets stuffed with new incoming requests while the existing threads are waiting for the API (external web service) to respond. I found a page that belonged to ASP.NET 1 (2004) that explained what are the optimized values for MaxIoThreads configuration item. (http://msdn.microsoft.com/en-us/library/ms998549.aspx) . I believe these values have changed in .NET 4. Does anyone know what are the new values/formulas for the process model settings? – Aref Karimi Jan 27 '14 at 23:33