How to dump goroutine stacktraces?

Question

I have Java background, and I love to use signal QUIT to inspect Java thread dump.

How to let Golang print out all goroutines stack trace?

Does this answer your question? [How to get the stacktrace of a panic (and store as a variable)](https://stackoverflow.com/questions/52103182/how-to-get-the-stacktrace-of-a-panic-and-store-as-a-variable) — hlin117, Jul 30 '20 at 20:31
You can `kill -ABRT ` to any Go process to kill and get a goroutine stack trace dump. — ahmet alp balkan, Sep 23 '21 at 18:06

score 163 · Accepted Answer · edited Jul 18 '15 at 12:27

163

To print the stack trace for the current goroutine, use PrintStack() from runtime/debug.

PrintStack prints to standard error the stack trace returned by Stack.

For example:

import(
   "runtime/debug"
)
...    
debug.PrintStack()

To print the stack trace for all goroutines use Lookup and WriteTo from runtime/pprof.

func Lookup(name string) *Profile
// Lookup returns the profile with the given name,
// or nil if no such profile exists.

func (p *Profile) WriteTo(w io.Writer, debug int) error
// WriteTo writes a pprof-formatted snapshot of the profile to w.
// If a write to w returns an error, WriteTo returns that error.
// Otherwise, WriteTo returns nil.

Each Profile has a unique name. A few profiles are predefined:

goroutine - stack traces of all current goroutines
heap - a sampling of all heap allocations
threadcreate - stack traces that led to the creation of new OS threads
block - stack traces that led to blocking on synchronization primitives

For example:

pprof.Lookup("goroutine").WriteTo(os.Stdout, 1)

edited Jul 18 '15 at 12:27

Scott C Wilson

19,102
10
61
83

answered Sep 30 '13 at 12:40

Intermernet

18,604
4
49
61

1

Does it print stack trace of all goroutines? – Sep 30 '13 at 12:50
It should, it calls `Stack`. "Stack returns a formatted stack trace of the goroutine that calls it. For each routine, it includes the source line information and PC value, then attempts to discover, for Go functions, the calling function or method and the text of the line containing the invocation." – Intermernet Sep 30 '13 at 12:53
1

Sorry, it prints the current goroutine stack trace only. – Oct 01 '13 at 02:48
4

@HowardGuo I've added an example using runtime/pprof to dump all stack traces. – Intermernet Oct 01 '13 at 03:16
1

thank you very much. may I suggest using `WriteTo(os.Stdout, 1)` instead of 0? so that stack trace information will contain actual symbol instead of memory address. – Oct 02 '13 at 22:50
2

I think this only outputs each thread's currently running goroutine, not *all* goroutines, ex: http://play.golang.org/p/0hVB0_LMdm – rogerdpack Jul 11 '14 at 20:28
@rogerdpack it says "goroutine - stack traces of all current goroutines". operative word being "current". – Intermernet Jul 15 '14 at 11:22
Using `2` instead of `1` for the second argument (`debug`) of `WriteTo()` will give you the same style trace as you get with `SIGQUIT`/`panic`, which I find more useful. – Danek Duvall Apr 30 '20 at 21:23

score 50 · Answer 2 · edited Apr 03 '18 at 19:29

There is an HTTP frontend for the runtime/pprof package mentioned in Intermernet's answer. Import the net/http/pprof package to register an HTTP handler for /debug/pprof:

import _ "net/http/pprof"
import _ "net/http"

Start an HTTP listener if you do not have one already:

go func() {
    log.Println(http.ListenAndServe("localhost:6060", nil))
}()

Then point a browser to http://localhost:6060/debug/pprof for a menu, or http://localhost:6060/debug/pprof/goroutine?debug=2 for a full goroutine stack dump.

There are other fun things you can learn about your running code this way too. Check out the blog post for examples and more details: http://blog.golang.org/profiling-go-programs

i made it run by it only shows the goroutines executed as far I see. Is there any way I could see all the "methods" that are executed after the main.go launches? — Lukas Lukac, Mar 22 '18 at 19:51

Rodolfo Carvalho · Answer 3 · 2019-09-27T19:36:19.760

46

Similar to Java, SIGQUIT can be used to print a stack trace of a Go program and its goroutines.
A key difference, however, is that by default sending SIGQUIT to Java programs do not terminate them, while Go programs do exit.

This approach requires no code change to print a stack trace of all goroutines of existing programs.

The environment variable GOTRACEBACK (see documentation of the runtime package) controls the amount of output generated. For example, to include all goroutines, set GOTRACEBACK=all.

The printing of the stack trace is triggered by an unexpected runtime condition (unhandled signal), originally documented in this commit, making it available since at least Go 1.1.

Alternatively, if modifying source code is an option, see other answers.

Note that in a Linux terminal, SIGQUIT can be conveniently sent with the key combination Ctrl+\.

edited Sep 27 '19 at 19:36

answered Feb 09 '16 at 10:55

Rodolfo Carvalho

1,737
1
20
18

6

While looking through docs I didn't find any mentions of SIGQUIT, rather SIGABRT. From my own tests (with go 1.7) the latter also worked over the former. – soltysh Apr 19 '17 at 15:36
4

this should be the top answer. – Steven Soroka Mar 01 '19 at 00:04
The docs refer to "when a Go program fails due to an unrecovered panic or an unexpected runtime condition". An uncaught signal (SIGQUIT, etc) is one of the latter. Why did I mention SIGQUIT? Because the OP expresses their love for using SIGQUIT with Java, and this answer emphasizes the similarity. Rewording the answer to make it clearer. – Rodolfo Carvalho Sep 27 '19 at 18:09

score 43 · Answer 4 · edited Jun 24 '20 at 01:16

43

To mimic the Java behaviour of stack-dump on SIGQUIT but still leaving the program running:

go func() {
    sigs := make(chan os.Signal, 1)
    signal.Notify(sigs, syscall.SIGQUIT)
    buf := make([]byte, 1<<20)
    for {
        <-sigs
        stacklen := runtime.Stack(buf, true)
        log.Printf("=== received SIGQUIT ===\n*** goroutine dump...\n%s\n*** end\n", buf[:stacklen])
    }
}()

edited Jun 24 '20 at 01:16

Pang

9,564
146
81
122

answered Dec 10 '14 at 10:01

Bryan

11,398
3
53
78

4

I think this is what the author was really looking for- mimics what Java does when you send a kill -QUIT. One small change I had to make was to change the first line of the for() loop to: "<- sigs". In other words, just discard the signal after waiting on it. Recent versions of Go will not let you declare a variable without later using it. – George Armhold Mar 25 '15 at 13:17
@Bryan, are you willing to license this under BSD or other more-permissive terms additional to the CC-BY-SA 3.0 required by StackOverflow? – Charles Duffy Jan 21 '18 at 19:59
1

@CharlesDuffy you can find much the same thing here under Apache licence: https://github.com/weaveworks/weave/blob/404c7a1ad4ae6dde1ed1d2cb55f5832ecc72f222/weaver/main.go#L163 – Bryan Jan 22 '18 at 15:16
I slightly improved this to print debug if it receives an os.Interupt signal, if a second signal comes in quickly (< 1 second) it exits, https://play.golang.org/p/dWgWrDFBOth – Aug 19 '21 at 10:16

score 30 · Answer 5 · answered Oct 31 '13 at 17:11

30

You can use runtime.Stack to get the stack trace of all goroutines:

buf := make([]byte, 1<<16)
runtime.Stack(buf, true)
fmt.Printf("%s", buf)

From the documentation:

func Stack(buf []byte, all bool) int

Stack formats a stack trace of the calling goroutine into buf and returns the number of bytes written to buf. If all is true, Stack formats stack traces of all other goroutines into buf after the trace for the current goroutine.

answered Oct 31 '13 at 17:11

Robert Kajic

8,689
4
44
43

This includes backtraces from *all* goroutines, nice! – rogerdpack Jul 11 '14 at 20:39
Is this the format that an unrecovered panic boils down to use? – Ztyx Dec 29 '14 at 10:00
3

Don't forget to add string(buf) or you'll print the raw bytes there. – koda May 18 '15 at 18:21
3

Maybe I'm doing something wrong, or perhaps the functionality has changed, but this doesn't retrieve anything for me except an empty slice of bytes? – 17xande Nov 03 '16 at 12:42
1

@koda there is no need to do `string(buf)` here, `fmt.Printf("%s", buf)` and `fmt.Printf("%s", string(buf))` do the exact same thing (see docs for `fmt` package); the only difference here is that the `string` version will copy the bytes from `buf` needlessly – kbolino Feb 10 '18 at 17:54
1

note that `runtime.Stack` returns the number of bytes it actually wrote inside `buf` and so you should manually slice `buf` to that length when printing it, to avoid writing a bunch of 0 bytes (which may be ignored by terminals but will show up in a file that output is redirected to) – kbolino Feb 10 '18 at 18:04
Note that the zero-initialization is necessary here. Trying to create a buffer with `make([]byte, 0, bufsize)` doesn't write any bytes, as the runtime does a length check instead of a capacity check. https://github.com/golang/go/blob/f4722d84499cc07fe8c8beb9b3154e59b7d21adf/src/runtime/mprof.go#L853 – typesanitizer Mar 01 '22 at 20:59

score 29 · Answer 6 · answered May 03 '17 at 08:57

29

Press CTRL+\

(If you run it in a terminal and just want to kill your program and dump the go routines etc)

I found this question looking for the key sequence. Just wanted a quick and easy way to tell if my program is leaking go routines :)

answered May 03 '17 at 08:57

Øyvind Skaar

2,278
15
15

1

This is why I love Go so much. It has all these features that make it way better than C or C++. – user1766438 May 26 '22 at 13:16
3

@user1766438 Ctrl+\ is the default terminal key sequence for SIGQUIT and has nothing to do with golang – phuclv May 27 '22 at 04:23
2

@phuclv yes but a C++ -compiled program doesn't print the stack trace when receiving SIGQUIT – arainone Jan 25 '23 at 10:31

score 14 · Answer 7 · answered Aug 26 '15 at 17:46

14

On *NIX systems (including OSX) send a signal abort SIGABRT:

pkill -SIGABRT program_name

answered Aug 26 '15 at 17:46

Kyle Kloepper

1,652
14
8

Apparently, sending SIGQUIT to a Java process [does not terminate it](https://docs.oracle.com/cd/E19455-01/806-1367/6jalj6mv1/index.html) like SIGABRT will. – Dave C Aug 26 '15 at 19:24
I found this to be the simplest and most matching solution to the original question. Often you need a stacktrace right away, without changing your code. – jotrocken Apr 07 '17 at 15:34

score 10 · Answer 8 · edited Feb 14 '21 at 18:06

By default, press ^\ keys ( CTRL+\ ) to dump the stack traces of all goroutines.

Otherwise, for more granular control, you can use panic. The simple way as of Go 1.6+:

go func() {
    s := make(chan os.Signal, 1)
    signal.Notify(s, syscall.SIGQUIT)
    <-s
    panic("give me the stack")
}()

Then, run your program like so:

# Press ^\ to dump the stack traces of all the user-created goroutines
$ GOTRACEBACK=all go run main.go

If you also want to print go runtime goroutines:

$ GOTRACEBACK=system go run main.go

Here are all the GOTRACEBACK options:

GOTRACEBACK=none omits the goroutine stack traces entirely.
GOTRACEBACK=single (the default) behaves as described above.
GOTRACEBACK=all adds stack traces for all user-created goroutines.
GOTRACEBACK=system is like all but adds stack frames for run-time functions and shows goroutines created internally by the run-time.
GOTRACEBACK=crash is like system but crashes in an operating system-specific manner instead of exiting. For example, on Unix systems, the crash raises SIGABRT to trigger a core dump.

Here is the documentation

The GOTRACEBACK variable controls the amount of output generated when a Go program fails due to an unrecovered panic or an unexpected runtime condition.

By default, a failure prints a stack trace for the current goroutine, eliding functions internal to the run-time system, and then exits with exit code 2. The failure prints stack traces for all goroutines if there is no current goroutine or the failure is internal to the run-time.

For historical reasons, the GOTRACEBACK settings 0, 1, and 2 are synonyms for none, all, and system, respectively.

The runtime/debug package's SetTraceback function allows increasing the amount of output at run time, but it cannot reduce the amount below that specified by the environment variable. See https://golang.org/pkg/runtime/debug/#SetTraceback.

score 5 · Answer 9 · edited Aug 02 '16 at 13:35

5

It's necessary to use the length returned by runtime.Stack() to avoid printing a bunch of empty lines after your stack trace. The following recovery function prints a nicely formatted trace:

if r := recover(); r != nil {
    log.Printf("Internal error: %v", r))
    buf := make([]byte, 1<<16)
    stackSize := runtime.Stack(buf, true)
    log.Printf("%s\n", string(buf[0:stackSize]))
}

edited Aug 02 '16 at 13:35

answered Jun 19 '15 at 17:28

David Tootill

541
5
11

There is no `runtime.Trace`; `runtime.Stack` [was already mentioned a year and a half ago](http://stackoverflow.com/a/19712747/55504). – Dave C Jun 19 '15 at 18:52
I've never seen that; what platform are you running on? – Bryan Nov 15 '15 at 10:10
What is it you haven't seen? The code should run on all platforms; I've tested it on Windows 7, Ubuntu 14.04, and Mac. – David Tootill Nov 16 '15 at 16:41
Never seen empty lines. – Bryan Jan 13 '17 at 17:10

score 1 · Answer 10 · edited Nov 11 '21 at 11:28

1

You can use this:

kill -3 YOUR_PROCESS_PID_ID

edited Nov 11 '21 at 11:28

Dharman

30,962
25
85
135

answered Nov 11 '21 at 05:59

M. Gopal

404
4
17

How to dump goroutine stacktraces?

10 Answers10

Linked

Related