I have a situation where I am working with a multithreaded C++ program in CentOS (Linux) and am trying to figure out which portion of the program is the "CPU resource hog". I have already identified the offending thread using the "top" command from the console. Now my question is, how can I find out what part of the code within that thread is hogging the CPU? Can you recommend any debugging tools / methods that would aid in this task? If it could tell me the line(s) that are being executed the most within the program, that should be helpful.
I would like to see if I can perform optimization on the offending code, and / or see if something is not happening correctly. The process itself appears to work correctly except for the fact that it slows down and hogs the CPU more than I would think it should.
I have looked into valgrind's tools a bit, and have not been able to produce anything helpful yet (although valgrind has a lot of tools). I tried the helgrind tool so far, and have looked at http://valgrind.org/docs/manual/manual.html for further guidance. It appears the callgrind tool may be useful if I can figure out how to use it. If anyone can tell me procedurally how to do debug what portions of code are being "over utilized" with valgrind, or point me to a good resource, it would be greatly appreciated.