Failed to compile bindsnoop eBPF c code outside of kernel tree

Question

I'm trying to build a BPF program written in C into the bpf bytecode. The program C code was taken from https://github.com/iovisor/bcc/blob/master/tools/bindsnoop.py (I used C code from bpf_text with some customizations). I didn't want to use bcc python tools, my purpose is to write both kernel and user space code in C, and I would like to build a kernel program outside of the kernel tree.

I use Ubuntu 20.10 and here is uname -a output: Linux bb-VirtualBox 5.8.0-63-generic #71-Ubuntu SMP Tue Jul 13 15:59:12 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Linux headers are installed, but for some reason I have two directories containing linux headers for 5.8.0-63: /usr/src/linux-headers-5.8.0-63 and /usr/src/linux-headers-5.8.0-63-generic

I used this post to create a makefile for my program: https://blogs.oracle.com/linux/post/bpf-in-depth-building-bpf-programs so the makefile looks like below:

OBJS = kprobe_bindsnoop_kern.o

LLC ?= llc
CLANG ?= clang
INC_FLAGS = -nostdinc -isystem `$(CLANG) -print-file-name=include`
EXTRA_CFLAGS ?= -O2 -emit-llvm

linuxhdrsgen ?= /usr/src/linux-headers-5.8.0-63-generic

LINUXINCLUDE =  -I$(linuxhdrsgen)/arch/x86/include \
                -I$(linuxhdrsgen)/arch/x86/include/generated/uapi \
                -I$(linuxhdrsgen)/arch/x86/include/generated \
                -I$(linuxhdrsgen)/arch/x86/include/uapi \
                -I$(linuxhdrsgen)/arch/x86/include/generated/uapi \
                -I$(linuxhdrsgen)/include/uapi \
                -I$(linuxhdrsgen)/include/generated/uapi \
                -I$(linuxhdrsgen)/include \
                -include ${linuxhdrsgen}/include/linux/kconfig.h

all: $(OBJS)

.PHONY: clean

clean:
    rm -f $(OBJS)

INC_FLAGS = -nostdinc -isystem `$(CLANG) -print-file-name=include`

$(OBJS):  %.o:%.c
    $(CLANG) $(INC_FLAGS) \
        -D__KERNEL__ -D__ASM_SYSREG_H \
        -Wno-unused-value -Wno-pointer-sign \
        -Wno-compare-distinct-pointer-types \
        -Wno-gnu-variable-sized-type-not-at-end \
        -Wno-address-of-packed-member -Wno-tautological-compare \
        -Wno-unknown-warning-option \
        -I../include $(LINUXINCLUDE) \
        $(EXTRA_CFLAGS) -c $< -o -| $(LLC) -march=bpf -filetype=obj -o $@

And kprobe_bindsnoop_kern.c starts from the following set of include files:

#include <uapi/linux/ptrace.h>
#pragma clang diagnostic push
#pragma clang diagnostic ignored "-Wtautological-compare"
#include <net/sock.h>
#pragma clang diagnostic pop
#include <net/inet_sock.h>
#include <net/net_namespace.h>
#include <bcc/proto.h>

Below you can see the compilation output:

clang -nostdinc -isystem `clang -print-file-name=include` \
    -D__KERNEL__ -D__ASM_SYSREG_H \
    -Wno-unused-value -Wno-pointer-sign \
    -Wno-compare-distinct-pointer-types \
    -Wno-gnu-variable-sized-type-not-at-end \
    -Wno-address-of-packed-member -Wno-tautological-compare \
    -Wno-unknown-warning-option \
    -I../include -I/usr/src/linux-headers-5.8.0-63-generic/arch/x86/include/ -I/usr/src/linux-headers-5.8.0-63-generic/arch/x86/include/generated/uapi -I/usr/src/linux-headers-5.8.0-63-generic/arch/x86/include/generated/ -I/usr/src/linux-headers-5.8.0-63-generic/arch/x86/include/uapi -I/usr/src/linux-headers-5.8.0-63-generic/arch/x86/include/generated/uapi -I/usr/src/linux-headers-5.8.0-63-generic/include/uapi -I/usr/src/linux-headers-5.8.0-63-generic/include/generated/uapi -I/usr/src/linux-headers-5.8.0-63-generic/include -include /usr/src/linux-headers-5.8.0-63-generic/include/linux/kconfig.h \
    -O2 -emit-llvm -c kprobe_bindsnoop_kern.c -o -| llc -march=bpf -filetype=obj -o kprobe_bindsnoop_kern.o
In file included from kprobe_bindsnoop_kern.c:22:
In file included from /usr/src/linux-headers-5.8.0-63-generic/include/uapi/linux/ptrace.h:143:
In file included from /usr/src/linux-headers-5.8.0-63-generic/arch/x86/include/asm/ptrace.h:5:
In file included from /usr/src/linux-headers-5.8.0-63-generic/arch/x86/include/asm/segment.h:6:
/usr/src/linux-headers-5.8.0-63-generic/arch/x86/include/asm/alternative.h:59:2: error: unknown type name 's32'
        s32 instr_offset;       /* original instruction */
        ^
/usr/src/linux-headers-5.8.0-63-generic/arch/x86/include/asm/alternative.h:60:2: error: unknown type name 's32'
        s32 repl_offset;        /* offset to replacement instruction */
        ^
/usr/src/linux-headers-5.8.0-63-generic/arch/x86/include/asm/alternative.h:61:2: error: unknown type name 'u16'
        u16 cpuid;              /* cpuid bit set for replacement */
        ^
/usr/src/linux-headers-5.8.0-63-generic/arch/x86/include/asm/alternative.h:62:2: error: unknown type name 'u8'
        u8  instrlen;           /* length of original instruction */
        ^
/usr/src/linux-headers-5.8.0-63-generic/arch/x86/include/asm/alternative.h:63:2: error: unknown type name 'u8'
        u8  replacementlen;     /* length of new instruction */
        ^
/usr/src/linux-headers-5.8.0-63-generic/arch/x86/include/asm/alternative.h:64:2: error: unknown type name 'u8'
        u8  padlen;             /* length of build-time padding */
        ^
/usr/src/linux-headers-5.8.0-63-generic/arch/x86/include/asm/alternative.h:85:8: error: unknown type name 'bool'
extern bool skip_smp_alternatives;
       ^
In file included from kprobe_bindsnoop_kern.c:22:
In file included from /usr/src/linux-headers-5.8.0-63-generic/include/uapi/linux/ptrace.h:143:
In file included from /usr/src/linux-headers-5.8.0-63-generic/arch/x86/include/asm/ptrace.h:5:
In file included from /usr/src/linux-headers-5.8.0-63-generic/arch/x86/include/asm/segment.h:169:
In file included from /usr/src/linux-headers-5.8.0-63-generic/arch/x86/include/asm/cache.h:5:
In file included from /usr/src/linux-headers-5.8.0-63-generic/include/linux/linkage.h:7:
In file included from /usr/src/linux-headers-5.8.0-63-generic/include/linux/export.h:43:
In file included from /usr/src/linux-headers-5.8.0-63-generic/include/linux/compiler.h:266:
/usr/src/linux-headers-5.8.0-63-generic/include/linux/kasan-checks.h:16:15: error: unknown type name 'bool'
static inline bool __kasan_check_read(const volatile void *p, unsigned int size)
              ^
/usr/src/linux-headers-5.8.0-63-generic/include/linux/kasan-checks.h:18:9: error: use of undeclared identifier 'true'
        return true;
               ^
/usr/src/linux-headers-5.8.0-63-generic/include/linux/kasan-checks.h:20:15: error: unknown type name 'bool'
static inline bool __kasan_check_write(const volatile void *p, unsigned int size)
              ^
/usr/src/linux-headers-5.8.0-63-generic/include/linux/kasan-checks.h:22:9: error: use of undeclared identifier 'true'
        return true;
               ^
/usr/src/linux-headers-5.8.0-63-generic/include/linux/kasan-checks.h:34:15: error: unknown type name 'bool'
static inline bool kasan_check_read(const volatile void *p, unsigned int size)
              ^
/usr/src/linux-headers-5.8.0-63-generic/include/linux/kasan-checks.h:36:9: error: use of undeclared identifier 'true'
        return true;
               ^
/usr/src/linux-headers-5.8.0-63-generic/include/linux/kasan-checks.h:38:15: error: unknown type name 'bool'
static inline bool kasan_check_write(const volatile void *p, unsigned int size)
              ^
/usr/src/linux-headers-5.8.0-63-generic/include/linux/kasan-checks.h:40:9: error: use of undeclared identifier 'true'
        return true;
               ^
In file included from kprobe_bindsnoop_kern.c:22:
In file included from /usr/src/linux-headers-5.8.0-63-generic/include/uapi/linux/ptrace.h:143:
In file included from /usr/src/linux-headers-5.8.0-63-generic/arch/x86/include/asm/ptrace.h:5:
In file included from /usr/src/linux-headers-5.8.0-63-generic/arch/x86/include/asm/segment.h:169:
In file included from /usr/src/linux-headers-5.8.0-63-generic/arch/x86/include/asm/cache.h:5:
In file included from /usr/src/linux-headers-5.8.0-63-generic/include/linux/linkage.h:7:
In file included from /usr/src/linux-headers-5.8.0-63-generic/include/linux/export.h:43:
In file included from /usr/src/linux-headers-5.8.0-63-generic/include/linux/compiler.h:267:
/usr/src/linux-headers-5.8.0-63-generic/include/linux/kcsan-checks.h:148:67: error: unknown type name 'size_t'
static inline void __kcsan_check_access(const volatile void *ptr, size_t size,
                                                                  ^
/usr/src/linux-headers-5.8.0-63-generic/include/linux/kcsan-checks.h:164:53: error: unknown type name 'size_t'
kcsan_begin_scoped_access(const volatile void *ptr, size_t size, int type,
                                                    ^
/usr/src/linux-headers-5.8.0-63-generic/include/linux/kcsan-checks.h:184:65: error: unknown type name 'size_t'
static inline void kcsan_check_access(const volatile void *ptr, size_t size,
                                                                ^
In file included from kprobe_bindsnoop_kern.c:22:
In file included from /usr/src/linux-headers-5.8.0-63-generic/include/uapi/linux/ptrace.h:143:
In file included from /usr/src/linux-headers-5.8.0-63-generic/arch/x86/include/asm/ptrace.h:6:
In file included from /usr/src/linux-headers-5.8.0-63-generic/arch/x86/include/asm/page_types.h:7:
In file included from /usr/src/linux-headers-5.8.0-63-generic/include/linux/mem_encrypt.h:17:
In file included from /usr/src/linux-headers-5.8.0-63-generic/arch/x86/include/asm/mem_encrypt.h:15:
/usr/src/linux-headers-5.8.0-63-generic/include/linux/init.h:155:8: error: unknown type name 'bool'
extern bool rodata_enabled;
       ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.

I tried to add #include <linux/types.h> or #include <asm/types.h> but it doesn't help. I checked many posts, like this Error compiling eBPF C code out of kernel tree and this how to build BPF program out of the kernel tree but still failed to get answer. What am I missing? Any help will be greatly appreciated ))

These `s32` and `u8` and so on are likely defined somewhere in some bcc header. You may be able to replace them with `__s32`, `__u8` and the like, that you probably pull from the headers you already include. — Qeole, Jul 28 '21 at 08:16
@wildplasser thanks! Do you mean I need to add manually all typedefs for bool, size_t, u8, u16, etc? I don't think it is a good idea — Sheifa, Jul 28 '21 at 08:28
@Qeole I added the typdefs but it does not help - looks like I need to rewrite all typedefs from linux/types.h and more. It now complains about atomic_t, phys_addr_t, bool, true/false and so on. Seems like a neverending story ... — Sheifa, Jul 28 '21 at 08:46
You are probably missing an include file. Maybe conditionally. If you really cannot find&fix it, the only way is to write one yourself. (which isn't too hard) — wildplasser, Jul 28 '21 at 08:47
@wildplasser I am sure I am missing an include file or include dir, just wonder what it can be. I agree that writing the include file is usually not a big deal but not in this case. Once I define the missing typedefs I immediately get the next bunch of missing defines... And I wonder why it can be compiled as a part of bcc and fails to do the same outside. — Sheifa, Jul 28 '21 at 09:00
BPF stems from BSD, so it will assume the host/kernel to be BSD. Porting it to Linux is a non-trivial operation. (IIRC, you'll have to find the interface(s) and put them into promiscuous mode, for instance) — wildplasser, Jul 28 '21 at 09:39
What? No. The BPF mentioned here is eBPF and specific to Linux. Nothing to do anymore with BSD. — pchaigno, Jul 28 '21 at 09:52
@wildplasser here is the output after solving u8/u16/etc stuff: In file included from /usr/src/linux-headers-5.8.0-63-generic/include/net/sock.h:38: /usr/src/linux-headers-5.8.0-63-generic/include/linux/list.h:33:42: warning: declaration of 'struct list_head' will not be visible outside of this function [-Wvisibility] static inline void INIT_LIST_HEAD(struct list_head *list) /usr/src/linux-headers-5.8.0-63-generic/include/linux/list.h:35:17: error: incomplete definition of type 'struct list_head' WRITE_ONCE(list->next, list); — Sheifa, Jul 28 '21 at 10:12
I don't understand why it is so challenging. The issue is not related to the fact I want to use the code from the bcc. Suppose I just want to create my own program that attaches to bind system call. It still will need to include #include #include #include But I cannot compile it for some reason and don't find any clue — Sheifa, Jul 28 '21 at 10:17
Have you looked at the newer version in libbpf-tools/bindsnoop.bpf.c? It relies on libbpf instead and may be easier to port out of bcc. — Qeole, Jul 28 '21 at 13:22
Hi @Qeole, thanks for the idea, bindsnoop.bpf.c code is very helpful — Sheifa, Jul 29 '21 at 09:04

score 1 · Answer 1 · answered Jul 29 '21 at 07:47

Thanks to everyone who tried to help me! Finally the issue was sooooo simple - just needed to change an order of the include directories specified in the makefile.

Just moved "-I$(linuxhdrsgen)/include" to be the first in the list and it solved the issue. Meaning, now the list of include dirs looks as follows:

LINUXINCLUDE = -I$(linuxhdrsgen)/include
-I$(linuxhdrsgen)/arch/x86/include
-I$(linuxhdrsgen)/arch/x86/include/generated/uapi
-I$(linuxhdrsgen)/arch/x86/include/generated
-I$(linuxhdrsgen)/arch/x86/include/uapi
-I$(linuxhdrsgen)/arch/x86/include/generated/uapi
-I$(linuxhdrsgen)/include/uapi
-I$(linuxhdrsgen)/include/generated/uapi
-include ${linuxhdrsgen}/include/linux/kconfig.h

Failed to compile bindsnoop eBPF c code outside of kernel tree

1 Answers1