2

I am doing baremetal development on ARM and emulating Raspi 3 on QEMU. Below is my minimal assembly code :

.section ".text.boot"

.global _start
_start:
1:  wfe
    b 1b

Below is my linker script :

SECTIONS
{
    . = 0x80000;
    .text : {*(.text.boot)}

    /DISCARD/ : { *(.comment) *(.gnu*) *(.note*) *(.eh_frame*) }
}

Below is my Makefile :

CC = /opt/gcc-arm-10.3-2021.07-x86_64-aarch64-none-elf/bin/aarch64-none-elf
CFLAGS = -Wall -O2 -ffreestanding -nostdinc -nostartfiles -nostdlib -g

all: clean kernel8.img

start.o: start.S
    ${CC}-gcc $(CFLAGS) -c start.S -o start.o

kernel8.img: start.o
    ${CC}-ld -nostdlib start.o -T link.ld -o kernel8.elf
    ${CC}-objcopy -O binary kernel8.elf kernel8.img

clean:
    rm kernel8.elf kernel8.img *.o >/dev/null 2>/dev/null || true

Now from one terminal, I am loading my kernel8.elf like below :

$ /opt/qemu-6.2.0/build/qemu-system-aarch64 -M raspi3b -kernel kernel8.elf -display none -S -s

From another terminal, I connect my gdb :

$ /opt/gcc-arm-10.3-2021.07-x86_64-aarch64-none-elf/bin/aarch64-none-elf-gdb ./kernel8.elf -ex 'target remote localhost:1234' -ex 'break *0x80000' -ex 'continue'
(gdb) info threads
  Id   Target Id                    Frame
  1    Thread 1.1 (CPU#0 [running]) _start () at start.S:6
  2    Thread 1.2 (CPU#1 [running]) _start () at start.S:5
* 3    Thread 1.3 (CPU#2 [running]) _start () at start.S:5
  4    Thread 1.4 (CPU#3 [running]) _start () at start.S:5

My cores are OK in this case, as all the 4 cores are running my assembly code. Upon continue the cores randomly hit breakpoints, which is perfect.

However, if I use the kernel8.img (objcopy binary output) instead of kernel8.elf, I see that only Core 1 is running my assembly, but other 3 cores seem to be stuck. Upon continue only Core 1 repeatedly hits breakpoint everytime.

(gdb) info threads
  Id   Target Id                    Frame
* 1    Thread 1.1 (CPU#0 [running]) _start () at start.S:5
  2    Thread 1.2 (CPU#1 [running]) 0x0000000000000300 in ?? ()
  3    Thread 1.3 (CPU#2 [running]) 0x0000000000000300 in ?? ()
  4    Thread 1.4 (CPU#3 [running]) 0x0000000000000300 in ?? ()

I tried set scheduler-locking on and continue on other 3 cores, but they seem to be stuck.

Why the kernel8.img is not working as kernel8.elf? I expected all ARM cores to be running the same code on reset, (as is happening with kernel8.elf) but its not happening with kernel8.img.

Naveen
  • 7,944
  • 12
  • 78
  • 165

1 Answers1

4

The QEMU -kernel option treats the file it loads differently depending on whether it is an ELF file or not.

If it is an ELF file, it is loaded according to what the ELF file says it should be loaded as, and started by executing from the ELF entry point. If it is not an ELF file, it is assumed to be a Linux kernel, and started in the way that the Linux kernel's booting protocol requires.

In particular, for a multi-core board, if -kernel gets an ELF file it starts all the cores at once at the entry point. If it gets a non-ELF file then it will do whatever that hardware is supposed to do for loading a Linux kernel. For raspi3b this means emulating the firmware behaviour of "secondary cores sit in a loop waiting for the primary core to release them by writing to a 'mailbox' address. This is the behaviour you're seeing in gdb -- the 0x300 address that cores 1-3 are at is in the "spin in a loop waiting" code.

In general, unless your guest code is a Linux kernel or is expecting to be booted in the same way as a Linux kernel, don't use the -kernel option to load it. -kernel is specifically "try to do what Linux kernels want", and it also tends to have a lot of legacy "this seemed like a useful thing to somebody" behaviour that differs from board to board or between different guest CPU architectures. The "generic loader" is a good way to load ELF files if you want complete manual control for "bare metal" work.

For more info on the various QEMU options for loading guest code, see this answer.

Peter Maydell
  • 9,707
  • 1
  • 19
  • 25
  • This is such an awesome answer and the one linked, answered by you is phenomenal. Answers every single doubt I had. Thank you @Peter – Naveen Jan 06 '22 at 15:52
  • One small clarification please. I tried `-M raspi3b -device loader,file=./kernel8.img,addr=0x80000,force-raw=on` but all of my cores seem to be stuck at address : `0x0000000000000200` – Naveen Jan 06 '22 at 16:05
  • Same behaviour observed when using `-M raspi3b -device loader,file=./kernel8.elf`. – Naveen Jan 06 '22 at 16:20
  • That is because when you load a file via the generic loader you get the behaviour of the CPU out of reset, which is to say it starts at address 0x0. You haven't put any code there, so it instantly takes an exception and then sits in a loop taking exceptions. You want your bare-metal image to include an exception vector table. – Peter Maydell Jan 06 '22 at 17:47
  • As per my understanding, in case of physical Raspi3, the GPU starts first which resets the CPU. This GPU, also puts some jump code at 0x0 (those magical bootcode.bin and start.elf files), so that processor executes (kernel) image placed at 0x80000. If this understanding is correct, in case of `-device loader` we are missing out that jump code (placed by GPU). Is that correct? – Naveen Jan 07 '22 at 10:46
  • But in case of `-kernel kernel8.elf`, I do all zeroes when I do `x/100xb 0x0`. This confuses me because in this case also ARM cores must have started execution at address 0x0. – Naveen Jan 07 '22 at 10:47
  • QEMU's raspi models don't implement the weird "GPU starts first" that the real hardware does. What you get is a simple "Arm CPU starts execution at the 0x0 reset vector in the usual way for Arm CPUs". For your other question, -kernel foo.elf honours the ELF entry point, but the generic loader does not (unless you use the cpu-num suboption -- but note that then you would need to sort out explicit entry points for all CPUs). – Peter Maydell Jan 07 '22 at 11:16