6

Hy there!

I'm running IIS7 on a Windows 2008 server.
On peek times we have following behaviour:

  • CPU load is near idle
  • Requests get queued (monitored with Resource Monitor)
  • Exeuction time gets over 10sec

1-4) Please see previous versions and edits

5) Doing stuff async

As suggested, I've created a simple web ... one page ... with this codebehind:

using System;
using System.Threading;
using System.Web;
using System.Web.UI;

namespace PerformanceTest
{
    public partial class AsyncPage : Page
    {
        protected override void OnInit(EventArgs e)
        {
            base.OnInit(e);

            var pageAsyncTask = new PageAsyncTask(this.BeginAsyncOperation, this.EndAsyncOperation, this.TimeoutAsyncOperation, null);
            this.RegisterAsyncTask(pageAsyncTask);
            // or
            //this.AddOnPreRenderCompleteAsync(this.BeginAsyncOperation, this.EndAsyncOperation);

            // this might be useful for doing cleanup or sth alike
            this.PreRenderComplete += HandlePreRenderComplete;
        }

        private void HandlePreRenderComplete(object sender, EventArgs e)
        {
            this.Trace.Write("HandlePreRenderComplete");
            this.Trace.Write(string.Format("managedThreadId #{0}", Thread.CurrentThread.ManagedThreadId));
        }

        private delegate void Sleep(int miliseconds);

        private IAsyncResult BeginAsyncOperation(object sender, EventArgs e, AsyncCallback asyncCallback, object state)
        {
            this.Trace.Write("BeginAsyncOperation");
            this.Trace.Write(string.Format("managedThreadId #{0}", Thread.CurrentThread.ManagedThreadId));

            var sleep = new Sleep(Thread.Sleep);
            return sleep.BeginInvoke(1000, asyncCallback, state);
        }

        private void EndAsyncOperation(IAsyncResult asyncResult)
        {
            this.Trace.Write("EndAsyncOperation");
            this.Trace.Write(string.Format("managedThreadId #{0}", Thread.CurrentThread.ManagedThreadId));
        }

        private void TimeoutAsyncOperation(IAsyncResult asyncResult)
        {
            this.Trace.Write("TimeoutAsyncOperation");
            this.Trace.Write(string.Format("managedThreadId #{0}", Thread.CurrentThread.ManagedThreadId));
        }
    }
}

Seems quite good, doesn't it? Actually it changes nothing, as stressing for more than 30 seconds, response time goes up to 8 seconds with CPU usage near 0%.

6) Updated machine.config

<system.net>
    <connectionManagement>
        <add address="*" maxconnection="12" />
    </connectionManagement>
</system.net>
<system.web>
    <processModel autoConfig="true"
                  maxWorkerThreads="100"
                  maxIoThreads="100"
                  minWorkerThreads="50"
                  minIoThreads="50" />
    <httpRuntime minFreeThreads="88"
                 minLocalRequestFreeThreads="76" />

</system.web>
  • Utilizing async would benefit performance as the threads would be returned to the request pool and you would have a higher degree of parallelism. This would mean serious reprogramming though. – Mikael Östberg Sep 28 '11 at 08:46
  • @MikaelÖstberg I know - that's why I've edited my question. This would just be a short-shot ... The current peak might get even worse :( ... Async pages would be the correct way of implementing such a page, but I'm not the creator - just the changer :) –  Sep 28 '11 at 08:49

4 Answers4

3

http://geekswithblogs.net/StuartBrierley/archive/2009/09/30/tuning-iis---machine.config-settings.aspx

Neville Kuyt
  • 29,247
  • 1
  • 37
  • 52
  • I've used some of the settings - though not the threads. I guess your question was "where can I change this"; the answer is "in machine.config". The link shows you which settings to change; I'm not convinced of the wisdom. I'd expect you to be able to move N higher, but the consequences of going to N+1 could well be far nastier than they are right now... – Neville Kuyt Sep 28 '11 at 11:03
2

You use a Page, and probably Session, so every page load lock all the other because of the Session. ! So you may call it Async, or call the PageAsyncTask, but the page is lock all the other calls because of the Session, and the page calls are get in a line to execute.

Disable the Session just for test, and then if you not use any session data, keep it disable for this page.

Look also: Replacing ASP.Net's session entirely

Web Garder

You can setup a web garden, that is give more pools to the same web application. By do that you need to re-check all your code and include synchronization with mutex, or database lock, because with two or more pools the same data can be access and write by different threads.

http://msdn.microsoft.com/en-us/library/aa479328.aspx

Community
  • 1
  • 1
Aristos
  • 66,005
  • 16
  • 114
  • 150
1

The answer to this is quite easy:

Don't block the thread

The fact of the matter is that IIS and an ASP.NET AppDomain is only able to handle N amount of concurrent requests. You can increase this number, but having a complete lock on thousands of concurrently running threads is a bottleneck nightmare. Without knowing exactly what's causing the ASP.NET page to respond in less than one second it's hard to suggest any performance tweaks, but the problem here is probably not with IIS, but in the code.

If the code is blocking a thread for several seconds without actually doing anything (as the CPU usage is witness to), there is some kind of IO that is so slow it should obviously be made asynchronously. A web-server can server an almost infinite amount of concurrent requests (only limited by available hardware resources) if these requests don't block threads. If they block threads, it can only perform as many requests as there are available threads, which has a hard upper limit.

Do stuff asynchronously

Rewrite your code so it doesn't lock the thread by invoking Begin... and End... method instead of their synchronous siblings. The Async CTP can help with wrapping these Begin and End calls in code that looks synchronous, but the performance benefit here looks to be so huge that you should consider rewriting whatever code it is that is locking up the page even without the async stuff.

Asbjørn Ulsberg
  • 8,721
  • 3
  • 45
  • 61
  • `The fact of the matter is that IIS and an ASP.NET AppDomain is only able to handle N amount of concurrent requests.` where to adapt this setting? –  Sep 28 '11 at 09:52
  • `Rewrite your code so it doesn't lock the thread by invoking Begin... and End... method instead of their synchronous siblings.` this will open a new thread to process the task, won't it? how much threads can the AppDomain handle? –  Sep 28 '11 at 09:54
  • thanks for your answer! i would like to respond: as mentioned i've adapted my code with `Thread.Sleep()` to ensure an execution-time of 1 sec (closed scope). so, if IIS can handle number N requests (and process them), each of them, which get actively processed, should not take longer than 1 second (plus some noise ...). the problem is, when doing more than N requests, they get queued, and the execution takes X (queue-time) + 1 sec. if i would/could increase the number of concurrent threads/requests there wouldn't be a problem :) –  Sep 28 '11 at 09:59
  • 2
    Just increasing the number of concurrent threads will only take you so far. The real problem is that each request is locking the thread while waiting for IO. Instead of locking the thread, Begin will release the thread so it can be used by other requests and End will only be invoked for the nanosecond you need to process the result when it is ready. – Asbjørn Ulsberg Sep 28 '11 at 11:14
  • I guess the .NET framework uses `WaitHandle` under the cover when you invoke `Begin...`, but that's implementation details I'm unfamiliar with. :-) – Asbjørn Ulsberg Sep 28 '11 at 13:00
  • thanks for your input, but please see my edit :) ... nothing changed –  Sep 29 '11 at 07:10
1

We had a similar problem once. It turned out that it was not our code that was causing the problem.

In our case the problem was with the TCP Chimney offload in windows 2008 R2. Windows 2008 trys to be clever by offloading work to the network card, but if the network card gets maxed out things slow down, but all the normal performance values show little load.

http://www.iislogs.com/steveschofield/troubleshooting-iis-7-network-performance-issues-and-tcp-chimney-offload-receive-side-scaling-and-network-direct-memory-access

Shiraz Bhaiji
  • 64,065
  • 34
  • 143
  • 252