How to trace a go function with BPF (BCC)

Question

I have some go-code (myclient). It looks somehow like this (here only the mypackage is relevant):

package mypackage

import (
...

func (o *Client) CreateUser(ctx context.Context, user *User, ...) (User, error) {
    ...
    if err != nil ...
      return nil, err
    ...
    return &user, nil
}

I run go build -o build/myclient cmd/myclient/main.go to get the binary.

λ bob [~]  → objdump -t ~/go/src/github.com/Corp/myclient/build/myclient | grep CreateUser
....
000000000087d490 l     F .text  0000000000000e55              github.com/Corp/myclient.(*Client).CreateUser

Now this binary is used by some other go programm and gets loaded.

Somehow like this:

bob      745322  0.1  0.1 195556  33172 pts/1    Sl+  20:18   0:07 some-prog
bob      750774  0.0  0.0 1229316 14188 pts/1    Sl+  20:22   0:00 /home/bob/go/src/github.com/Corp/myclient/build/myclient

Now I try to use BCC Tools for tracing/debug. The reason for this is that some strange errors happens in production and it is important to view the args and return values of the function above. I was inspired by this: http://www.brendangregg.com/blog/2017-01-31/golang-bcc-bpf-function-tracing.html

When using funccount I see this here (after invoking the function by some call to the some-prog)

λ bob [~]  → sudo funccount '/home/bob/go/src/github.com/Corp/myclient/build/myclient:*.CreateUser'
Tracing 5 functions for "b'/home/bob/go/src/github.com/Corp/myclient/build/myclient:*.CreateUser'"... Hit Ctrl-C to end.
^C
FUNC                                                    COUNT
...
b'github.com/Corp/myclient.(*Client).CreateUser'        1
Detaching...

But when I try to use trace (or gotrace) it looks like this:

λ bob [~]  → trace '/home/bob/go/src/github.com/Corp/myclient/build/myclient:mypackage.*'  
could not determine address of symbol b'mypackage.*'

λ bob [~]  → trace '/home/bob/go/src/github.com/Corp/myclient/build/myclient:github.com/Corp/myclient.CreateUser'                          
could not determine address of symbol b'github.com/Corp/myclient.CreateUser'

λ bob [~]  → trace '/home/bob/go/src/github.com/Corp/myclient/build/myclient:github.com/Corp/myclient.(*Client).CreateUser'
error in probe '/home/bob/go/src/github.com/Corp/myclient/build/myclient:github.com/Corp/myclient.(*Client).CreateUser': expected format string in "s

λ bob [~]  → trace '/home/bob/go/src/github.com/Corp/myclient/build/myclient:*.CreateUser'                                                               
could not determine address of symbol b'*.CreateUser'

Is there any possibility to trace such a go-function (especially inspecting struct args and return value) with BCC tooling or eBPF/(bpftrace?) (without adding any code)

score 0 · Answer 1 · answered Jan 10 '20 at 11:05

trace is a python script, you can open it in vim and then search for a function named _parse_probe. In this function, line 96:

(spec, sig, rest) = re.match(r'([^ \t\(]+)(\([^\(]*\))?(.*)', text).groups()

A golang function symbol with '(' character is splitted into 3 parts, whereas a normal function will match the first regular substring. You can modify the regular expression--delete the '(' part, and then trace will work. Good luck.

This is actually a good answer. Changed the `([^ \t\(]+)` part to `([^ \t]+)`, and could trace the specific golang function. — ron, Sep 11 '21 at 21:08

How to trace a go function with BPF (BCC)

1 Answers1