1

I'm trying to profile a program that performs IO operations (such as network device). Here are some certain facts:

1) My workload generator saturates the request.

2) My IO device is not fully utilized (around 70%)

3) top shows that there my CPU core is having idle time of 30%.

What I don't understand is that since the request is saturated at the workload generator, if there is available CPU resource and device resource, shouldn't it be fully utilized?

The source code that I'm working on is extremely large so need to analyze the hot-path. Currently, all I can think of the possible reason is:

A) There maybe some "wait" routine in the hot-path.

Because, by the fact that CPU idle time is around 30%, the hot-path is not always executed on CPU core.

NOTE: Based on the observation on each CPU core, there is no bottleneck in a core, so the problem is not about load-balancing. All cores are in idle with 30%.

I wish if some of you may have similar experience and could give me some hint for constructing my hypothesis for the bottleneck analysis.

jaeyong
  • 8,951
  • 14
  • 50
  • 63
  • While you're waiting, [*try this method.*](http://stackoverflow.com/a/378024/23771) It shows you where "bottlenecks" are, regardless of I/O or whatever. – Mike Dunlavey Mar 26 '15 at 22:30
  • See, I would say you should not construct a hypothesis about what takes time. (I dislike the term "bottleneck" because it implies the problem is a location in the code.) The most important thing you can do is to simply make no assumptions at all about what the problem is. Instead, let the diagnostic tool or technique tell you what it is. The reason is, if you could guess what the problem is, you would have already fixed it. What often happens is - people guess what will be a problem, fix it, and then find out the problem was elsewhere, and they actually slowed it down, by fixing a non-problem. – Mike Dunlavey Mar 27 '15 at 14:43
  • Once you get experience doing this, you see certain kinds of problems tend to recur, and then you can make guesses, but you also know you should never act on those guesses. You should let the tool or technique tell you what the problem is, regardless of your prior guesses. – Mike Dunlavey Mar 27 '15 at 14:48

0 Answers0