How to debug an eBPF program that uses cilium/ebpf to write a go user program?

Question

Thanks to those who came to check it out.

Now I'm having some problems. The ebpf program I was facing was an eBPF program written in go's cilium/ebpf library. There was no problem compiling, but there was a problem with the runtime and the bpf program would not load. The error is as follows:

root@ubuntu:/home/golang/go/src/xdp-nat# ./xdp-nat ens33 lo
2023/07/24 23:57:49 loading objects: field XdpNatInner2outerFunc: program xdp_nat_inner2outer_func: load program: permission denied: 554: (71) r1 = *(u8 *)(r8 +17): R8 invalid mem access ' inv' (506 line(s) omitted)

I wanted to debug it, but I found that when running with Go, I would report an error and couldn't find the symbols used in the cilium library.

golang@ubuntu:~/go/src/xdp-nat$ go run main.go
# command-line-arguments
./main.go:197:10: undefined: bpfObjects
./main.go:198:12: undefined: loadBpfObjects

So how should I debug an EBPF program if something goes wrong? He is a problem with the internal loading, I don't know the details. According to the error, I found 71 lines of the EBPF source code and found that it was just a member variable of the struct. What could be the problem?

root@ubuntu:/home/golang/go/src/xdp-nat# ./xdp-nat ens33 lo
if1 ok
if2 ok
2023/07/25 00:58:31 loading objects: field XdpNatInner2outerFunc: program xdp_nat_inner2outer_func: load program: permission denied: 554: (71) r1 = *(u8 *)(r8 +17): R8 invalid mem access ' inv' (506 line(s) omitted)

I think it might be a problem with the access of a member variable, but I can't find it, and I won't debug it a little bit, I don't know what the problem is. Some attempts are already on top.

Try to find if it is a BPF verify error. Using `bpftool prog load` to test the bpf file — seamaner, Jul 26 '23 at 07:29

score 1 · Answer 1 · answered Jul 27 '23 at 11:16

I wanted to debug it, but I found that when running with Go, I would report an error and couldn't find the symbols used in the cilium library.

Your project uses bpf2go a tool provided by cilium/ebpf that generates types and functions based on your BPF code. The bpfObjects and loadBpfObjects symbols are located in separate go files.

When you call go run main.go it only includes your main.go file, not the generated files, go run . should solve your issue.

2023/07/25 00:58:31 loading objects: field XdpNatInner2outerFunc: program xdp_nat_inner2outer_func: load program: permission denied: 554: (71) r1 = *(u8 *)(r8 +17): R8 invalid mem access ' inv' (506 line(s) omitted)

This indicates your BPF program doesn't pass the verifier. You can dump the omitted lines with the following error check:

if err != nil {
  var verr *ebpf.VerifierError
  if errors.As(err, &verr) {
    fmt.Printf("%+v\n", verr)
  }
}

Reading the verifier log can be tricky, if you get stuck I recommend opening another question, make sure to post at least the last 50-100 lines of the log and the sources if you can.

How to debug an eBPF program that uses cilium/ebpf to write a go user program?

1 Answers1