11

I have a process written in golang that runs as a daemon.

After a few days it stops producing output with no apparent reason. I think it might be an internal deadlock. If I do a strace -p <PID> I can see the line

futex(0x9aaba0, FUTEX_WAIT, 0, NULL

and apparently that call never ends. I would like to use the already running process (since I don't know how/when to trigger the bug again) to debug the problem.

How can I see what goroutines are running and where they? How can I dump from the process whatever other information might be useful to debug the problem?

juanleon
  • 9,220
  • 30
  • 41
  • 1
    Very related: http://stackoverflow.com/questions/23354810/how-can-i-dump-all-a-go-processs-stacks and http://stackoverflow.com/questions/19094099/how-to-dump-goroutine-stacktraces – siritinga Feb 24 '15 at 09:20
  • I think, without recompiling with some debug output, this will be pretty difficult. From https://golang.org/doc/gdb#Inspecting_the_stack : "GDB does not understand Go programs well." ... "As a consequence, although GDB can be useful in some situations, it is not a reliable debugger for Go programs, particularly heavily concurrent ones. Moreover, it is not a priority for the Go project to address these issues, which are difficult. In short, the instructions below should be taken only as a guide to how to use GDB when it works, not as a guarantee of success." – Intermernet Feb 24 '15 at 11:39
  • http://stackoverflow.com/questions/19094099/how-to-dump-goroutine-stacktraces may be useful – rogerdpack Feb 24 '15 at 17:49

2 Answers2

21

You can send any go program a QUIT signal, and it will exit with a full stack trace, showing the state of all goroutines, and how they are blocked.

As for strace, go programs are always multithreaded, so you always need to add the -f option

JimB
  • 104,193
  • 13
  • 262
  • 255
  • 1
    Unfortunately, the daemon captures the QUIT signal to do some cleanup and then calling `exit(0)`. Good point with the `-f` option. I am afraid that I will need to recompile the daemon with some added debugging capabilities, and then hope for the bug to happen again :-( – juanleon Feb 24 '15 at 15:50
2

Here are some methods from Deadlocks: the dark side of concurrency (covers what a deadlock is, how to debug, and and how to avoid them)

  • Use SIGQUIT
    • This produces a nice backtrace on Unix based system
    • Send it with kill -QUIT pid
    • Or press CTRL-\
    • Does not work on Windows
  • Use the http debug handler
    • Retrieve the backtrace with a web browser or curl
    • Work on all platforms
Community
  • 1
  • 1
zangw
  • 43,869
  • 19
  • 177
  • 214