15

I'm debugging an issue in a multi-threaded linux process, where a certain thread appears to not execute for few seconds. Looking at strace output revealed it waits for futex e.g.
1673109 14:36:28.600329 futex(0x44b8d20, FUTEX_WAIT_PRIVATE,
1673109 14:36:33.221850 <... futex resumed> ) = 0 <4.621514>

How can I find out out what this futex(0x44b8d20) refers to in user space, i.e. how to map this to a locking construct which is using futex internally.

Aman Jain
  • 10,927
  • 15
  • 50
  • 63

1 Answers1

13

I would use a simple systemtap script so that would help you to quickly find out addresses of contended futex locks. When I say address, I am referring to the first argument of futex() syscall.

  1. You can download the simple system tap script that finds the contended user space locks here:
    https://sourceware.org/systemtap/examples/process/futexes.stp

    If you have systemtap installed on your system,
    just start this system tap script: stap futexes.stp

  2. Capture the strace output as you did before.

  3. If you end the system tap script execution by simply doing Ctrl-C,
    you will get the output of contended futexes.

  4. Finally in your strace output,
    search for futex calls in which the second argument (operation type) is FUTEX_WAIT.
    For example : futex(0x7f58a31999d0, FUTEX_WAIT, 4508, NULL) = 0

  5. Then you can search for the first argument in system tap script output.
    Something like : ome[4489] lock 0x7f58a31999d0 contended 1 times, 7807 avg us

If you look at this system tap script,
it nicely prints the process name and process/thread id for you.
This makes it easy to find what you are looking for.

However one note is that executing the systemtap script will actually hook a syscall system wide.

TheCodeArtist
  • 21,479
  • 4
  • 69
  • 130
Akin Ocal
  • 471
  • 4
  • 11
  • This is really smart, but unfortunately I'm using OS X. Thanks to your post I've now started investigating using DTrace with Ruby probes though. That's definitely an avenue I hadn't even thought of exploring before. – Reck Aug 07 '16 at 14:40
  • great answer! Sadly i seemed to have an issue with it as authored on ubuntu lts 14.04.05 `sudo ./futexes.stp semantic error: while resolving probe point: identifier 'syscall' at ./futexes.stp:15:7 source: probe >syscall.futex { semantic error: no match Pass 2: analysis failed. [man error::pass2] Number of similar error messages suppressed: 1. Rerun with -v to see them. ` – ThorSummoner Nov 01 '18 at 19:24
  • 1
    @ThorSummoner You need to run `futexes.stp` with the `systemtap` utility. On Ubuntu this can be installed with `apt-get install systemtap` and then executed (as root) `stap futexes.stp`. More info here: https://sourceware.org/systemtap/SystemTap_Beginners_Guide/using-systemtap.html – dpoggi Dec 27 '18 at 20:10