-1

Consider the normal scenario where an ASP.NET Core Web API application executes the service Controller action, but instead of executing all the work under the same thread (thread pool thread) until the response is created, I would like to use non-pooled threads (ideally pre-created) to execute the main work, either by scheduling one of these threads from the initial action pooled thread and free the pooled thread for serving other incoming requests, or passing the job to a pre-created non-pooled thread.

Among other reasons, the main reason to have these non-pooled and long running threads is that some requests may be prioritized and their threads put on hold (synchronized), thus it would not block new incoming requests to the API due to thread pool starvation, but older requests on hold (non-pooled threads) may be waked up and rejected and some sort of call back to the thread pool to return the web response back to the clients.

In summary, the ideal solution would be using a synchronization mechanism (like .NET RegisterWaitForSingleObject) where the pooled thread would hook to the waitHandle but be freed up for other thread pool work, and a new non-pooled thread would be created or used to carry on the execution. Ideally from a list of pre-created and idle non-pooled threads.

Seems async-await only works with Tasks and threads from the .NET thread pool, not with other threads. Also most techniques to create non-pooled threads do not allow the pooled thread to be free and return to the pool.

Any ideas? I'm using .NET Core and latest versions of tools and frameworks.

Pedro Oliveira
  • 140
  • 1
  • 8
  • `ideally pre-created` in other words, pooled. What you describe is what ASP.NET already does. Each request is served by a *different* thread that comes from a pool of pre-created threads. There's no blocking, no starvation. Long running tasks can be performed with `await Task.Run(..)` *without* blocking the original thread. Not that it would help at all, you are still using *a* thread. Might as well use the original – Panagiotis Kanavos Jul 03 '18 at 15:54
  • The only case when you want a "different" thread is when making an asynchronous call, eg to another service, database or performing IO. In this case there's nothing for the thread to do except wait. Instead of waiting though, you can use asynchronous methods like `HttpClient.GetStringAsync` or `ExecuteReaderAsync` etc. In this case *no* thread is used. The pooled thread goes back to the pool and is available for other requests. When the async call completes a thread is pulled from the pool (not the original one) to process the response and execute the rest of the call – Panagiotis Kanavos Jul 03 '18 at 15:57
  • What is the *actual* problem you want to solve? What you described is how the TPL and ASP.NET already works. It even has work-stealing so an idle thread can "steal" work from a busy thread's queue. This sounds like a case of [the XY Problem](https://meta.stackexchange.com/questions/66377/what-is-the-xy-problem). You encountered a problem X and assumed Y is the solution. When Y didn't work you asked for Y, not X – Panagiotis Kanavos Jul 03 '18 at 16:00
  • Thanks for your comments Panagiotis. Pre-created means they are running threads in idle state being served by queues. These threads will perform a vast number of operations and checks on each request and run service logic, but should have already been instantiated. Meaning that there will not be the normal overhead or scheduling a new thread and instantiating all internal objects. After finishing a request, a thread can start right away on another one queued. The thread pool threads are assigned to work but have to re-create the whole context for each request. Makes sense? – Pedro Oliveira Jul 03 '18 at 16:13
  • One of the problem is that I could expect bursts of 100's of requests that may take several seconds to execute or the server may put on hold according to a prioritization engine, and one way to avoid thread pool starvation and deal with slow increase of thread pool size by ASP.NET, would be to move on to a larger list of pre-created and ready to run non-pooled threads. – Pedro Oliveira Jul 03 '18 at 16:18
  • 2
    I repeat, that's **exactly** what the thread pool is. That's exactly what the pooled threads are. If you worry about how fast the threadpool can grow you can increase its minimum size. If you worry about *blocking* just don't block. If you want to use different requests with different priorities, a) this probably means that you need two different services, not one and b) there are mechanisms to handle that already. Whether it's a different TaskScheduler or `ActionBlock` instances with different DOP settings or emulating awaiting with TaskCompletionSource – Panagiotis Kanavos Jul 03 '18 at 16:28
  • Perhaps you should check Stephen Cleary's article [Async Programming : Introduction to Async/Await on ASP.NET](https://msdn.microsoft.com/en-us/magazine/dn802603.aspx?f=255&MSPPError=-2147217396) from the October 2014 issue of MSDN Magazine – Panagiotis Kanavos Jul 03 '18 at 16:36
  • In this post it also mentions that messing with thread pool priorities or similarly, running a permanent foreground thread is ill advised with modern OS https://stackoverflow.com/a/31045819/3254405 – boateng Jul 03 '18 at 16:38
  • @numbtongue Did not mention changing thread priorities, but choosing which requests are served first by the backends, based on priorities and other business rules, in which case potentially many will have to be put on hold. Reason why doing this outside of the thread pool would be advantageous to have full control. – Pedro Oliveira Jul 03 '18 at 16:50

1 Answers1

1

Thank you for the comments provided. The suggestion to check TaskCompletionSource was fundamental. So my goal was to have potentially hundreds or thousands of API requests on ASP.NET Core and being able to serve only a portion of them at a given time frame (due to backend constraints), choosing which ones should be served first and hold the others until backends are free or reject them later. Doing all this with thread pool threads is bad: blocking/holding and having to accept thousands in short time (thread pool size growing).

The design goal was the request jobs to move their processing from the ASP.NET threads to non pooled threads. I plan to to have these pre-created in reasonable numbers to avoid the overhead of creating them all the time. These threads implement a generic request processing engine and can be reused for subsequent requests. Blocking these threads to manage request prioritization is not a problem (using synchronization), most of them will not use CPU at all time and the memory footprint is manageable. The most important is that the thread pool threads will only be used on the very start of the request and released right away, to be only be used once the request is completed and return a response to the remote clients.

The solution is to have a TaskCompletionSource object created and passed to an available non-pooled thread to process the request. This can be done by queuing the request data together with the TaskCompletetionSource object on the right queue depending the type of service and priority of the client, or just passing it to a newly created thread if none available. The ASP.NET controller action will await on the TaskCompletionSouce.Task and once the main processing thread sets the result on this object, the rest of the code from the controller action will be executed by a pooled thread and return the response to the client. Meanwhile, the main processing thread can either be terminated or go get more request jobs from the queues.

using System;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Mvc;

namespace MyApi.Controllers
{
    [Route("api/[controller]")]
    public class ValuesController : Controller
    {
        public static readonly object locker = new object();
        public static DateTime time;
        public static volatile TaskCompletionSource<string> tcs;

        // GET api/values
        [HttpGet]
        public async Task<string> Get()
        {
            time = DateTime.Now;
            ShowThreads("Starting Get Action...");

            // Using await will free the pooled thread until a Task result is available, basically
            // returns a Task to the ASP.NET, which is a "promise" to have a result in the future.
            string result = await CreateTaskCompletionSource();

            // This code is only executed once a Task result is available: the non-pooled thread 
            // completes processing and signals (TrySetResult) the TaskCompletionSource object
            ShowThreads($"Signaled... Result: {result}");
            Thread.Sleep(2_000);
            ShowThreads("End Get Action!");

            return result;
        }

        public static Task<string> CreateTaskCompletionSource()
        {
            ShowThreads($"Start Task Completion...");

            string data = "Data";
            tcs = new TaskCompletionSource<string>();

            // Create a non-pooled thread (LongRunning), alternatively place the job data into a queue
            // or similar and not create a thread because these would already have been pre-created and
            // waiting for jobs from queues. The point is that is not mandatory to create a thread here.
            Task.Factory.StartNew(s => Workload(data), tcs, 
                CancellationToken.None, TaskCreationOptions.LongRunning, TaskScheduler.Default);

            ShowThreads($"Task Completion created...");

            return tcs.Task;
        }

        public static void Workload(object data)
        {
            // I have put this Sleep here to give some time to show that the ASP.NET pooled
            // thread was freed and gone back to the pool when the workload starts.
            Thread.Sleep(100);

            ShowThreads($"Started Workload... Data is: {(string)data}");
            Thread.Sleep(10_000);
            ShowThreads($"Going to signal...");

            // Signal the TaskCompletionSource that work has finished, wich will force a pooled thread 
            // to be scheduled to execute the final part of the APS.NET controller action and finish.
            // tcs.TrySetResult("Done!");
            Task.Run((() => tcs.TrySetResult("Done!")));
            // The only reason I show the TrySetResult into a task is to free this non-pooled thread 
            // imediately, otherwise the following line would only be executed after ASP.NET have 
            // finished processing the response. This briefly activates a pooled thread just execute 
            // the TrySetResult. If there is no problem to wait for ASP.NET to complete the response, 
            // we do it synchronosly and avoi using another pooled thread.

            Thread.Sleep(1_000);

            ShowThreads("End Workload");
        }

        public static void ShowThreads(string message = null)
        {
            int maxWorkers, maxIos, minWorkers, minIos, freeWorkers, freeIos;

            lock (locker)
            {
                double elapsed = DateTime.Now.Subtract(time).TotalSeconds;

                ThreadPool.GetMaxThreads(out maxWorkers, out maxIos);
                ThreadPool.GetMinThreads(out minWorkers, out minIos);
                ThreadPool.GetAvailableThreads(out freeWorkers, out freeIos);

                Console.WriteLine($"Used WT: {maxWorkers - freeWorkers}, Used IoT: {maxIos - freeIos} - "+
                                  $"+{elapsed.ToString("0.000 s")} : {message}");
            }
        }
    }
}

I have placed the whole sample code so anyone can easily create as ASP.NET Core API project and test it without any changes. Here is the resulting output:

MyApi> Now listening on: http://localhost:23145
MyApi> Application started. Press Ctrl+C to shut down.
MyApi> Used WT: 1, Used IoT: 0 - +0.012 s : Starting Get Action...
MyApi> Used WT: 1, Used IoT: 0 - +0.015 s : Start Task Completion...
MyApi> Used WT: 1, Used IoT: 0 - +0.035 s : Task Completion created...
MyApi> Used WT: 0, Used IoT: 0 - +0.135 s : Started Workload... Data is: Data
MyApi> Used WT: 0, Used IoT: 0 - +10.135 s : Going to signal...
MyApi> Used WT: 2, Used IoT: 0 - +10.136 s : Signaled... Result: Done!
MyApi> Used WT: 1, Used IoT: 0 - +11.142 s : End Workload
MyApi> Used WT: 1, Used IoT: 0 - +12.136 s : End Get Action!

As you can see the pooled thread runs until the await on the TaskCompletionSource creation, and by the time the Workload starts to process the request on the non-pooled thread there is ZERO ThreadPool threads being used and remains using no pooled threads for the entire duration of the processing. When the Run.Task executes the TrySetResult fires a pooled thread for a brief moment to trigger the rest of the controller action code, reason the Worker thread count is 2 for a moment, then a fresh pooled thread runs the rest of the ASP.NET controller action to finish with the response.

Pedro Oliveira
  • 140
  • 1
  • 8