I recently used Mike Dunlavey's "Poor Man's Profiling" (How can I profile C++ code running on Linux? ) to find some surprising sources of overhead in my code. Yay.
To summarize, run your program in a debugger and manually capture some backtraces. If you find the same stack traces keep showing up, you have a pretty good idea of where the overheads are in your program.
The "identify" step requires eyeballing the stack traces. Humans are pretty good at pattern matching but I'd like to automate this process a bit.
Surely there is a tool that can take N backtraces and show me which backtraces were in which routines?
There is something called Stack Trace Analysis Tool (https://github.com/LLNL/STAT) but it operates on executables itself -- very much in the High Performance Computing domain where you might wonder why/where a parallel (MPI) program is hung. Not quite what I'm looking for: I have already collected the backtraces.