1

We have a Windows Service which hosts a number of WCF services and, in an unrelated part of the app, makes extensive use of the TPL Task class to asynchronously do relatively short bits of work.

It is my understanding that WCF uses managed I/O threads from the ThreadPool to execute requests. I noticed that after deploying a feature which significantly raised the applications use of Tasks, and as such the use of ThreadPool worker threads as well, performance of a couple of web services has become very slow. We're talking minutes instead of less than a second. The number of Tasks actually trying to run at any one time can range between 20 and 1000, which makes me think that any new (last in) work needing some CPU time could be forced to wait for quite some time.

Does the (in my case extremely large) number of busy ThreadPool worker threads affect the ThreadPool's managed I/O threads? Could these two be connected in any way? Or should I start looking elsewhere...

Thanks!

EDIT: I just did a spot check on a live deployment of the app while trying to call one of the long running web services which is quite typical: 70 Tasks running, Windows Service process has about 150 threads, average CPU usage 30-60%. By typical I mean that CPU usage is generally around 40% and rarely exceeds 80%, however there are a large number of Tasks running and threads used by the process.

andrej351
  • 904
  • 3
  • 18
  • 38
  • Have you profiled the overall IO/CPU activity to see if you've simply pegged the box? – Marc Gravell Jan 11 '11 at 07:11
  • I also wonder if there are some complex issues involving blocking here, but that is very hard to investigate in isolation. – Marc Gravell Jan 11 '11 at 07:12
  • That is a lot of threads i.e. 150. If they are 'busy' threads and all contending for some time on a core, a lot of time is going to be wasted just thrashing to get scheduled. What are these threads doing? Try limiting max threads for TPL loops. – Tim Lloyd Jan 12 '11 at 01:27
  • I assume they would be responding to the enormous amount of TPL Tasks i'm starting. I'll try to run some tests with limited concurrency. – andrej351 Jan 12 '11 at 01:36
  • TPL tasks are queued, and the scheduler will try to keep the maximum number of threads down to a minimum. A figure of 150 is very high. Those 150 threads are fighting over a limited number of cores. – Tim Lloyd Jan 12 '11 at 01:43

2 Answers2

1

If your tasks spend any amount of time sleeping or waiting for I/O, you might benefit from starting your tasks with TaskCreationOptions.LongRunning. This causes the thread pool to create more threads, rather than waiting for queued threads to complete.

    task = Task.Factory.StartNew(() =>
   {
       DoLongRunningWork();
   }, TaskCreationOptions.LongRunning);

If you want to test the concept, there is an experiment here which you might find helpful.

Community
  • 1
  • 1
Greg Sansom
  • 20,442
  • 6
  • 58
  • 76
1

Note: I've never answered my own question before so i hope this is appropriate. Feel free to comment or disagree, but it is looking like this isn't getting answered by anyone else.

So, the problem causing our performance issues was something completely unrelated and is now solved.

After this was resolved, the web services began responding in a timely manner again. This leads me to conclude that you can have quite a lot of TPL Tasks running concurrently (like the hundreds in my example) and, even with their heavy usage of Thread Pool worker threads, WCF will still be able to use the Thread Pool's managed I/O threads efficiently to handle your web service requests.

andrej351
  • 904
  • 3
  • 18
  • 38