Requests are queuing in Azure AppService though it has enough threads in threadpool

Question

I have written an api using asp.net webapi and deployed it in azure as Appservice. Name of my controller is TestController and My action method is something like bellow.

    [Route("Test/Get")]
    public string Get()
    {
        Thread.Sleep(10000);
        return "value";
    }

So for each request it should wait for 10 sec before return string "value". I have also written another endpoint to see the number of threads in threadpool working for executing requests. That action is something like bellow.

    [Route("Test/ThreadInfo")]
    public ThreadPoolInfo Get()
    {
        int availableWorker, availableIO;
        int maxWorker, maxIO;

        ThreadPool.GetAvailableThreads(out availableWorker, out availableIO);
        ThreadPool.GetMaxThreads(out maxWorker, out maxIO);

        return new ThreadPoolInfo
        {
            AvailableWorkerThreads = availableWorker,
            MaxWorkerThreads = maxWorker,
            OccupiedThreads = maxWorker - availableWorker
        };
    }

Now when We make 29 get calls concurrently to Test/Get endpoint it takes almost 11 seconds to get succeed all requests. So server executes all the requests concurrently in 11 threads. To see the threads status, making call to Test/ThreadInfo right after making call to Test/Get returns immediately(without waiting) { "AvailableWorkerThreads": 8161, "MaxWorkerThreads": 8191, "OccupiedThreads": 30 }

Seems 29 threads are executing Test/Get requests and 1 thread is executing Test/ThreadInfo request.

When I make 60 get calls to Test/Get it takes almost 36 seconds to get succeed. Making call to Test/ThreadInfo(takes some time) returns { "AvailableWorkerThreads": 8161, "MaxWorkerThreads": 8191, "OccupiedThreads": 30 }

If we increase requests number, value of OccupiedThreads increases. Like for 1000 requests it takes 2 min 22 sec and value of OccupiedThreads is 129.

Seems request and getting queued after 30 concurrent call though lot of threads are available in WorkerThread. Gradually it increases thread for concurrent execution but that is not enough(129 for 1000 request).

As our services has lot of IO call(some of them are external api call and some are database query) the latency is also high. As we are using all IO calls async way so server can serve lot of request concurrently but we need more concurrency when processor are doing real work. We are using S2 service plan with one instance. Increasing instance will increase concurrency but we need more concurrency from single instance.

After reading some blog and documentation on IIS we have seen there is a setting minFreeThreads. If the number of available threads in the thread pool falls bellow the value for this setting IIS starts to queue request. Is there anything in appservice like this? And Is it really possible to get more concurrency from azure app service or we are missing some configuration there?

Answer 1

At last got the answer for my question. The thing is that The ASP.NET thread pool maintains a pool of threads that have already incurred the thread initialization costs and are easy to reuse. The .NET thread pool is also self-tuning. It monitors CPU and other resource utilization, and it adds new threads or trims the thread pool size as needed. When there are lot of requests and not enough thread in pool is available then thread pool starts to add new threads in the pool and before that it runs its own algorithm to see the status of memory and cpu use of the system which takes long amount of time and that is why we see slowly increase of worker thread in the pool and get lot of request queued. But luckly there is an option to set the number of worker thread before thread pool switches to an algorithm to add new thread. The code is something like bellow.

    public string Settings()
    {
        int minWorker, minIOC;
        ThreadPool.GetMinThreads(out minWorker, out minIOC);

        if (ThreadPool.SetMinThreads(300, minIOC)){ 
            return "The minimum number of threads was set successfully.";
        }
        else
        {
            return "The minimum number of threads was not changed.";
        }

    }

Here ThreadPool.SetMinThreads(300, minIOC) is setting the value of minimum threads threadpool will create before switching to an algorithm for adding or removing thread. I have Added this method as an action of my webapi controller and then after running this action by making a request when I made 300 concurrent request to Test/Get endpoint all was running and completed in 11 seconds and no request were queued.

Answer 2

Per my understanding, you need to check the MinWorkerThreads via ThreadPool.GetMinThreads which could retrieve the number of idle threads the ThreadPool maintains in anticipation of new requests. I would recommend that you return the current thread pool info after executed your action (Tetst/Get) instead of making a new request to Test/ThreadInfo. Based on your code, I tested it on my web app with the B1 pricing tier as follows:

1000 concurrent requests, sleep 15s for each action, overall elapsed 3 min 20 sec.

The thread pool creates and destroys worker threads in order to optimize throughput, if the current threads could handle the requests then no more threads would be created. Once those threads finish executing their activities, they are then returned to the thread pool.

Then, I used Asynchronous programming and changed the action as follows:

public async Task<ActionResult> DoJob(int id)
{
    await Task.Delay(TimeSpan.FromSeconds(15));
    return ThreadInfo();
}

Result:

Nearly 3000 concurrent requests:

In general, AvailableWorkerThreads just means the number of additional worker threads that can be started instead of worker threads have been created for you. I would recommend that you using Asynchronous programming and deploy your real work on azure web app, then check the real performance and find the related approaches if any bottlenecks.