How to access user space function arguments in bpf?

Question

I am trying to instrument a user space nginx function by using libbpf. I am able to attach a uprobe it, and print pid, tid and so on from the probe. However, I am having great issues whenever I try to parse function argument data. I have been able to do this with bpftrace but am unable to do so with libbpf. My question is, how to properly access and print arguments of the user space function I want to trace?

nginx.bpf.c

#include "ngx_http.h"
#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_core_read.h>

char LICENSE[] SEC("license") = "Dual BSD/GPL";

SEC("uprobe//usr/sbin/nginx:ngx_http_finalize_request")
int handle_ngx_http_finalize_request(struct ngx_http_request_s* r, ngx_int_t rc)
{
    u_char *s_ptr;
    u_char str[128];
    int err;

    err = bpf_probe_read_user(&s_ptr, sizeof(s_ptr), &r->request_line.data);
    if (!s_ptr || err < 0) {
        bpf_printk("Error %d\n", err);
        return -2;
    }

    bpf_probe_read_user_str(str, sizeof(str), &s_ptr);

    bpf_printk("String: %s\n", str);

    return 0;
}

Whenever I try to parse the function arguments the bpf_probe_read_user returns error -14. When I try to use bpf_core_read the verifier rejects the code with following error.

❯ sudo ./nginx
libbpf: loading object 'nginx_bpf' from buffer
libbpf: elf: section(3) uprobe//usr/sbin/nginx:ngx_http_finalize_request, size 280, link 0, flags 6, type=1
libbpf: sec 'uprobe//usr/sbin/nginx:ngx_http_finalize_request': found program 'handle_ngx_http_finalize_request' at insn offset 0 (0 bytes), code size 35 insns (280 bytes)
libbpf: elf: section(4) .reluprobe//usr/sbin/nginx:ngx_http_finalize_request, size 32, link 12, flags 40, type=9
libbpf: elf: section(5) license, size 13, link 0, flags 3, type=1
libbpf: license of nginx_bpf is Dual BSD/GPL
libbpf: elf: section(6) .rodata, size 22, link 0, flags 2, type=1
libbpf: elf: section(7) .BTF, size 12720, link 0, flags 0, type=1
libbpf: elf: section(9) .BTF.ext, size 252, link 0, flags 0, type=1
libbpf: elf: section(12) .symtab, size 240, link 1, flags 0, type=2
libbpf: looking for externs among 10 symbols...
libbpf: collected 0 externs total
libbpf: map 'nginx_bp.rodata' (global data): at sec_idx 6, offset 0, flags 80.
libbpf: map 0 is "nginx_bp.rodata"
libbpf: sec '.reluprobe//usr/sbin/nginx:ngx_http_finalize_request': collecting relocation for section(3) 'uprobe//usr/sbin/nginx:ngx_http_finalize_request'
libbpf: sec '.reluprobe//usr/sbin/nginx:ngx_http_finalize_request': relo #0: insn #13 against '.rodata'
libbpf: prog 'handle_ngx_http_finalize_request': found data map 0 (nginx_bp.rodata, sec 6, off 0) for insn 13
libbpf: sec '.reluprobe//usr/sbin/nginx:ngx_http_finalize_request': relo #1: insn #28 against '.rodata'
libbpf: prog 'handle_ngx_http_finalize_request': found data map 0 (nginx_bp.rodata, sec 6, off 0) for insn 28
libbpf: loading kernel BTF '/sys/kernel/btf/vmlinux': 0
libbpf: map 'nginx_bp.rodata': created successfully, fd=4
libbpf: sec 'uprobe//usr/sbin/nginx:ngx_http_finalize_request': found 1 CO-RE relocations
libbpf: prog 'handle_ngx_http_finalize_request': relo #0: <byte_off> [2] typedef ngx_http_request_t.request_line.data (0:21:1 @ offset 992)
libbpf: prog 'handle_ngx_http_finalize_request': relo #0: no matching targets found
libbpf: prog 'handle_ngx_http_finalize_request': relo #0: substituting insn #1 w/ invalid insn
libbpf: prog 'handle_ngx_http_finalize_request': BPF program load failed: Invalid argument
libbpf: prog 'handle_ngx_http_finalize_request': -- BEGIN PROG LOAD LOG --
R1 type=ctx expected=fp
; int handle_ngx_http_finalize_request(ngx_http_request_t* r, ngx_int_t rc)
0: (bf) r3 = r1
1: <invalid CO-RE relocation>
failed to resolve CO-RE relocation <byte_off> [2] typedef ngx_http_request_t.request_line.data (0:21:1 @ offset 992)
processed 2 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0
-- END PROG LOAD LOG --
libbpf: prog 'handle_ngx_http_finalize_request': failed to load: -22
libbpf: failed to load object 'nginx_bpf'
libbpf: failed to load BPF skeleton 'nginx_bpf': -22
Failed to open and load BPF skeleton

Here is bpftrace code that works.

nginx.bt

uprobe:/usr/sbin/nginx:ngx_http_finalize_request
{
    $req = (struct ngx_http_request_s*)arg0;
    printf("Request Line: %s\n", str($req->request_line.data));
}

So I would like to parse data and do some custom logic depending on it.

Here are the ngx_http_request_s and the ngx_str_t structs for the curious.

EDIT

User space code:

#include <stdio.h>
#include <unistd.h>
#include <sys/resource.h>
#include <bpf/libbpf.h>
#include "nginx.skel.h"

static int libbpf_print_fn(enum libbpf_print_level level, const char *format, va_list args)
{
    return vfprintf(stderr, format, args);
}

int main(int argc, char **argv)
{
    struct nginx_bpf *skel;
    int err;

    libbpf_set_strict_mode(LIBBPF_STRICT_ALL);
    libbpf_set_print(libbpf_print_fn);

    skel = nginx_bpf__open_and_load();
    if (!skel) {
        fprintf(stderr, "Failed to open and load BPF skeleton\n");
        return 1;
    }

    err = nginx_bpf__attach(skel);
    if (err) {
            fprintf(stderr, "Failed to auto-attach BPF skeleton: %d\n", err);
            goto cleanup;
    }

    printf("Successfully started!\n");

    for (;;) {
        sleep(1);
    }

cleanup:
    nginx_bpf__destroy(skel);
    return -err;
}

score 3 · Answer 1 · answered Dec 08 '22 at 22:00

3

You are actually not defining BPF program correctly. You should do something like the below. Note that you also shouldn't use CO-RE variants of bpf_core_read() or BPF_CORE_READ(), use BPF_PROBE_READ_USER() instead:

SEC("uprobe//usr/sbin/nginx:ngx_http_finalize_request")
int BPF_KPROBE(handle_ngx_http_finalize_request,
               struct ngx_http_request_s* r, ngx_int_t rc)
{
    u_char *s_ptr;
    u_char str[128];
    int err;

    /* you can access rc directly now, btw */

    s_ptr = BPF_PROBE_READ_USER(r, request_line.data);
    /* note no dereferencing of s_ptr above */
    bpf_probe_read_user_str(str, sizeof(str), s_ptr); 

    bpf_printk("String: %s\n", str);

    return 0;

You don't have to use BPF_PROBE_READ_USER() macro, you can do the same with just bpf_probe_read_user() like you did in your example. BPF_PROBE_READ_USER() will be especially handy if you need to follow few levels of pointers, though.

But there is no CO-RE for user-space types, it's only for kernel types, because kernel provides BTF information to allow relocating offsets properly.

answered Dec 08 '22 at 22:00

Andrii Nakryiko

136
1
3

Thanks for the reply. I am still unnable to actually access the contents of `ngx_http_request_s` struct. The output of `str` array is empty... Could this error be related to me omitting the calculation of the function offset in the user space code? I have update the original post with the user space code. I have omitted the offset calc due to the code being to seemingly load properly. – nela Dec 10 '22 at 16:05
what error does bpf_probe_read_user_str() return? if it's -EFAULT, then either s_ptr value is incorrect, or to get that memory requires page fault and uprobe programs don't allow faults. – Andrii Nakryiko Dec 12 '22 at 19:00
It is indeed -EFAULT. Considering our [previous discussion](https://github.com/libbpf/libbpf-bootstrap/discussions/133#discussioncomment-4365389), I am more inclined to believe that the s_ptr value is incorrect, as the bpf program is unable to read system-wide headers, and therefore gets the memory address wrong somehow. Also not entirely sure how a page fault might occur in this context, as the value I am trying to access is already on the stack. – nela Dec 13 '22 at 14:46
Revisiting this, in what way can the value be 'incorrect'? I've tried the same with `rc` where the type is defined as `typedef intptr_t ngx_int_t`, and getting the same results. – nela Dec 20 '22 at 19:21

How to access user space function arguments in bpf?

EDIT

1 Answers1