1

I have a multi-threaded web application with about 1000~2000 threads at production environment.

I expect CPU usage on w3wp.exe but System Idle Process eats CPU. Why?

Xaqron
  • 29,931
  • 42
  • 140
  • 205
  • 3
    There is rarely a good reason to have 1,000 threads. – SLaks May 12 '11 at 15:11
  • 1
    What are the threads *suppose* to be doing? – Brian Gideon May 12 '11 at 15:13
  • They are connecting to external resources and put data to a `shared buffer` then another thread will remove data and use it. There's a chance cpu usage is because of memory management by CLR since I had `LOH` problem and now I'm using `buffer manager`. After a peak in threads, the CPU problem appeared. – Xaqron May 12 '11 at 15:19

4 Answers4

4

The Idle process isn't actually a real process, it doesn't "eat" your CPU time. the %cpu you see next to it is actually unused %cpu (more or less).

The reason for the poor performance of your application is most likely due to your 2000 threads. Windows (or indeed any operating system) was never meant to run so many threads at a time. You're wasting most of the time just context switching between them, each getting a couple of milliseconds of processing time every ~30 seconds (15ms*2000=30sec!!!!).

Rethink your application.

Blindy
  • 65,249
  • 10
  • 91
  • 131
  • Good point about the overhead of context switching. (I'm aware it's a different case, but out of curiosity: wouldn't mainframes or supercomputers be designed to run thousands of threads? Wall-clock time is usually not a constraint there, so there may be fewer context switches per unit of time...) – Piskvor left the building May 12 '11 at 15:15
  • Quite the contrary, they wouldn't use threads at all. If I designed a mainframe that accepted thousands of jobs, I'd do them one by one in the order they were given to me (FIFO). That would ensure the minimum amount of CPU time wasted. Those applications are written to crunch A LOT of data, they won't be IO bound. – Blindy May 12 '11 at 15:29
  • Keep in mind that, IO bound issues aside, multithreading is made for humans to give them an impression of speed. You actually *lose* speed by forcing context switches, it just *seems* like it's faster because it started earlier. – Blindy May 12 '11 at 15:31
  • You are indeed quite correct; silly me, forgetting about serial processing altogether. – Piskvor left the building May 12 '11 at 15:44
1

the idle process is simply holding process time until a program needs it, its not actually eating any cycles at all. you can think the system idle time as 'available cpu'

Richard Smith
  • 272
  • 2
  • 3
  • Under `Task Manager` it shows 50%-70% CPU usage by `System Idle Process` where `w2wp` only uses 2%-5% – Xaqron May 12 '11 at 15:16
1

System Idle Process is not a real process, it represents unused processor time.

This means that your app doesn't utilize the processor completely - it may be memory-bound or CPU-bound; possibly the threads are waiting for each other, or for external resources? Context switching overhead could also be a culprit - unless you have 2000 cores, the threads are not actually running all at the same time, but assigned time slices by the task scheduler, this also takes some time.

Piskvor left the building
  • 91,498
  • 46
  • 177
  • 222
1

You have not provided a lot of details so I can only speculate at this point. I would say it is likely that most of those threads are doing nothing. The ones that are doing something are probably IO bound meaning that they are spending most of their waiting for the external resource to respond.

Now lets talk about the "1000~2000 threads". There are very few cases (maybe none) where having that many threads is a good idea. I think your current issue is a perfect example of why. Most of those threads are (apparently anyway) doing nothing but wasting resources. If you want to process multiple tasks in parallel, espcially if they are IO bound, then it is better to take advantage of pooled resources like the ThreadPool or by using the Task Parallel Library.

Brian Gideon
  • 47,849
  • 13
  • 107
  • 150
  • Actually they are threads from ASP.NET thread pool. Each two threads (a thread pair) are responsible for fulfilling a request. I was planning to use but changed design to dedicated threads for requests. Does moving to `HttpAsyncHandler` and lowering thread count helps me? – Xaqron May 12 '11 at 15:41
  • @Xaqron: Possibly. Could you try with different number of threads (10,50,100,200,300,...,3000) and measure if there's a significant difference in performance? – Piskvor left the building May 13 '11 at 07:58