If the system is really busy with IO, just look at top
and you'll see the IO-bound process usually stuck in a D-state.
strace -c myprog
is my best friend for a first attempt at all generic 'what is my application doing/where is it spending most time' questions. Strace can also attach to running processes, so you can observe the program as it's running.
Another good strace trick is to output it (with strace -o myprogrun.log
) to a log file , then view it with a modern vim
as it does a very nice job syntax highlighting the log. It's a lot easier to find things this way, as the default strace output is not very human-readable.
Important thing to remember is to log to another partition/set of disks than where the IO problem is! Do not induce extra IO problems as strace can generate a lot of output. I like to use a TmpFS or ZRAM RAM-disks for such occasions.