XDP program ip link error: Prog section rejected: Operation not permitted

Question

I try to get into XDP, for this I have this very small program:

// SPDX-License-Identifier: GPL-2.0
#include <linux/bpf.h>
#include "bpf/bpf_helpers.h"
#include "xdpsock.h"

struct {
    __uint(type, BPF_MAP_TYPE_ARRAY);
    __uint(max_entries, MAX_SOCKS);
    __uint(key_size, sizeof(int));
    __uint(value_size, sizeof(int));
} xsks_map SEC(".maps");

SEC("xdp_sock") int xdp_sock_prog(struct xdp_md *ctx) {

    return XDP_DROP;
}

But if I try to load it into a virtual interface veth-basic02, I get this error:

$ sudo ip -force link set dev veth-basic02 xdp object xdpsock_kern.o section xdp_sock

Prog section 'xdp_sock' rejected: Operation not permitted (1)! - Type: 6 - Instructions: 2 (0 over limit) - License:

Verifier analysis:

Error fetching program/map!

Kernel-Version: 5.3.0-28-generic

This is the Makefile I am using:

OBJS = xdpsock_kern.o

LLC ?= llc
CLANG ?= clang
INC_FLAGS = -nostdinc -isystem `$(CLANG) -print-file-name=include`
EXTRA_CFLAGS ?= -O2 -emit-llvm

# In case up-to-date headers are not installed locally in /usr/include,
# use source build.

linuxhdrs ?= /usr/src/linux-headers-5.1.0-050100

LINUXINCLUDE =  -I$(linuxhdrs)/arch/x86/include/uapi \
                -I$(linuxhdrs)/arch/x86/include/generated/uapi \
                -I$(linuxhdrs)/include/generated/uapi \
                -I$(linuxhdrs)/include/uapi \
                -I$(linuxhdrs)/include  \
                -I/bpf

prefix ?= /usr/local

INSTALLPATH = $(prefix)/lib/bpf

install_PROGRAM = install
install_DIR = install -dv

all:    $(OBJS)

.PHONY: clean

clean:
    rm -f $(OBJS)

INC_FLAGS = -nostdinc -isystem `$(CLANG) -print-file-name=include`

$(OBJS):  %.o:%.c
    $(CLANG) $(INC_FLAGS) \
                -D__KERNEL__ -D__ASM_SYSREG_H \
                -Wno-unused-value -Wno-pointer-sign \
                -Wno-compare-distinct-pointer-types \
                -Wno-gnu-variable-sized-type-not-at-end \
                -Wno-address-of-packed-member -Wno-tautological-compare \
                -Wno-unknown-warning-option \
                -I../include $(LINUXINCLUDE) \
                $(EXTRA_CFLAGS) -c $< -o -| $(LLC) -march=bpf -filetype=obj -o $@

install: $(OBJS)
    $(install_DIR) -d $(INSTALLPATH) ; \
    $(install_PROGRAM) $^ -t $(INSTALLPATH)

uninstall: $(OBJS)
    rm -rf $(INSTALLPATH)

Lockdown:

$ dmesg | grep Lockdown
[    1.283355] Lockdown: swapper/0: Hibernation is restricted; see man kernel_lockdown.7
[   11.313219] Lockdown: systemd: /dev/mem,kmem,port is restricted; see man kernel_lockdown.7
[   11.337794] Lockdown: systemd: BPF is restricted; see man kernel_lockdown.7
[   17.147844] Lockdown: Xorg: ioperm is restricted; see man kernel_lockdown.7

Edit: echo 1 > /proc/sys/kernel/sysrq + echo x > /proc/sysrq-trigger + Alt+SysRq+x indeed solves the problem - I can finally load the XDP-Program! Funny easter egg though. Thank you @Qeole!

Answer 1

eBPF: Operation not permitted

There are several possible causes for a permission error (-EPERM returned by bpf(), which you can observe with strace -e bpf <command>) when working with eBPF. But no so many. Usually, they fall under one of the following items:

• User does not have the required capabilities (CAP_SYS_ADMIN, CAP_NET_ADMIN, ... typically depending on the types of the programs being used). This is usually solved by running as root, who has all necessary capabilities. In your case you run with sudo, so you are covered.

• Creating the BPF object (new map, or loading a program) would exceed the limit for the amount of memory that can be locked in the kernel by the user. This is usually solved (for root) by using ulimit -l <something_big> in the terminal, or setrlimit() in a C program. Very unlikely in your case, your program is very small and you did not mention having a lot of BPF objects loaded on your system.

• There are a few more possibilies, like trying to write on maps that are “frozen” or read-only etc., or trying to use function calls for non-root users. These are usually for more advanced use cases and should not be hit with a program as simple as yours.

Lockdown, Secure Boot, EFI and (unfortunate) backports for bpf() restrictions

But the problem that you seem to be hitting could be related to something else. “Lockdown” is a security module that was merged into Linux 5.5 kernel. It aims at preventing users to modify the running Linux image. It turns out that several distributions decided to backport Lockdown to their kernels, and sometimes they picked patches that predated the final version that was merged to mainline Linux.

Ubuntu and Fedora, for example, have a bunch of custom patches to backport that feature to the kernels used in Disco/19.04 and Eoan/19.10 (kernel 5.3 for the latter, I don't remember for Disco). It includes a patch that completely disables the bpf() system call when Lockdown is activated, meaning that creating maps or loading BPF programs is not possible. Also, they enabled Lockdown by default when Secure Boot is activated, which, I think, is the default for machines booting with EFI.

See also this blog post: a good way to check if Lockdown is affecting your BPF usage is to try and load minimal programs, or to run dmesg | grep Lockdown to see if it says something like:

Lockdown: systemd: BPF is restricted; see man kernel_lockdown.7

So for Ubuntu 19.04 and 19.10, for example, you have to disable Lockdown to work with eBPF. This may be done with a physical stroke of the SysRq key + x (I have not tested), but NOT by writing to /proc/sysrq-trigger (Ubuntu disabled it for this operation). Alternatively, you can disable Secure Boot (in the BIOS or with mokutil, search for the relevant options on the Internet, and do not forget to check the security implications).

Note that Linux kernel 5.4 or newest has the mainline restrictions for bpf(), which do not deactivate the system call, so Focal/20.04 and newest will not be affected. Upgrading to a new kernel might thus be another workaround. I filed a ticket a few days ago to ask for this change to be backported (instead of deactivating bpf()) and the work is in progress, so by the time new readers look at the answer Lockdown impact on eBPF might well be mitigated (Edit: Should be fixed on Ubuntu 19.10 with kernel 5.3.0-43). Not sure how other distros handle this. And it will still have strong implications for tracing with eBPF, though.