3

We have found that several CPU intensive queries mean that our API server no longer responses to simple requests. The API server is an .net core application with kestrel, which is executed in a Kubernetes cluster. However, if the application is running on a Windows or Linux host, the task prioritization seems to work perfectly. The service is responding even if there are dozens of CPU intensive requests. So there seems to be a significant difference between the Docker environment and the host environment.

I use this API method for test purposes:

public void SimulateHighCpuLoad()
{
    var previousPriority = Thread.CurrentThread.Priority;

    try
    {
        Thread.CurrentThread.Priority = ThreadPriority.Lowest;

        var until = DateTime.Now.AddSeconds(30);
        var num = 0L;
        var random = new Random();

        // do senseless work for 30 seconds
        while (DateTime.Now < until)
        {
            num = (random.Next() + Environment.TickCount + num) % (random.Next(10000) + 1);
            num *= num++;
        }
    }
    finally
    {
        Thread.CurrentThread.Priority = previousPriority;
    }
}

My goal is to prioritize CPU intensive methods lower so that the application can always respond to other requests (such as health requests for the LivenessProbe). The Thread.Priority seems to be completely ignored within docker enviroment

Thomas
  • 153
  • 9
  • 1
    Increase the size of the thread pool, use a `SemaphoreSlim` to limit the number of CPU heavy tasks.... so that some threads are available to process I/O? – Jeremy Lakeman Jul 09 '20 at 07:06
  • Thank you @JeremyLakeman. I checked the thread pool min/max size for the host enviroment and the kubernetes enviroment. The values are equal. The ThreadPool.GetAvailableThreads is always > 30,000. – Thomas Jul 09 '20 at 09:35
  • How much CPU has been assigned to the environment? – mjwills Jul 14 '20 at 08:00
  • https://stackoverflow.com/a/51998044/34092 may assist. – mjwills Jul 14 '20 at 08:37
  • Where is the cluster deploy (cloud, on-prem)? Are there any resource limits on namespace/ cluster level? – kool Jul 21 '20 at 14:21
  • The cluster has 3 nodes with each 2 CPUs. The container has a limit of 1500m CPU. The cluster is deployed in an azure enviroment. – Thomas Jul 22 '20 at 19:49

1 Answers1

2

According to this forum thread, we are not the only ones with this problem. Thread scheduling seems to work fundamentally differently within a Docker container (or K8s cluster).

We tried a lot and were able to solve the problem with a mixture of throttling and Thread.Sleep(0).

Note the following documentation for Thread.Sleep

If the value of the millisecondsTimeout argument is zero, the thread relinquishes the remainder of its time slice to any thread of equal priority that is ready to run

Without Thread.Sleep(0), CPU-intensive threads cause that no other threads have been processed. No timers were called either. Setting a low thread priority is ineffective within a Docker container.

Thomas
  • 153
  • 9