4

I'm debugging an issue in a patch to PostgreSQL where a word in shared memory seems to get overwritten unintentionally.

Valgrind isn't any help as it can't keep track of interactions in shared memory between multiple processes.

The address that gets overwritten is fairly stable but not totally fixed, though it's always identified by a pointer in a global struct initialized early in startup by each process.

I'm trying to find a way to get a stack trace whenever any process writes to the address of interest, but it's proving rather harder than I would've expected.

gdb watchpoints aren't any help, as gdb can't follow fork() and establish the same watch on child processes. Manually doing it with multiple gdb processes is very cumbersome due to the number of child processes PostgreSQL uses and the timing issues involved in setting it up by hand.

perf userspace probes looked promising, but seem to attach only to functions, there's no apparent way to trap a write to a memory address.

So is there any way to grab stack traces for each writer to a given shared memory address across multiple processes?

Sourav Ghosh
  • 133,132
  • 16
  • 183
  • 261
Craig Ringer
  • 307,061
  • 76
  • 688
  • 778

1 Answers1

3

gdb can't follow fork() and establish the same watch on child processes

A sufficiently recent GDB can do that. Documentation here and here.

Employed Russian
  • 199,314
  • 34
  • 295
  • 362
  • 2
    Thankyou for the exceptionally helpful pointer. I thought I'd replied before, but apparently not. Specifically, `schedule-multiple on` and `detach-on-fork off` do the trick. Breakpoints, watchpoints, etc, are inherited by all child processes being debugged. Very impressive. [catchpoints on fork](https://sourceware.org/gdb/onlinedocs/gdb/Set-Catchpoints.html) are also very, very handy. – Craig Ringer Nov 26 '14 at 03:42
  • The only issue I'm running into with it is that [gdb likes to stop whenever any inferior exits](http://stackoverflow.com/q/27140941/398670). – Craig Ringer Nov 26 '14 at 05:02
  • I landed up [writing this up in a blog post for later](http://blog.2ndquadrant.com/processes-breakpoints-watchpoints-postgresql/). Returning to this topic, non-stop mode and schedule-multiple are useful. You'll also want `set print symbol-loading off`. – Craig Ringer Jun 21 '16 at 06:23