Getting a panic at populate_rootfs with bigger initramfs

Question

I'm using buildroot with initramfs. And it is working fine with uImages smaller than 14M.

But when I add packages and get bigger, this kernel panic occurs.

Kernel command line: noinitrd ramdisk_size=30720 console=ttyS0,115200n8 oops=panic panic=10 rdinit=/sbin/init mem=64M ubi.mtd=2 mtdparts=nand0:0x200000@0x0(u-boot),0x1E00000@l
PID hash table entries: 256 (order: -2, 1024 bytes)
Dentry cache hash table entries: 8192 (order: 3, 32768 bytes)
Inode-cache hash table entries: 4096 (order: 2, 16384 bytes)
Memory: 44656K/65536K available (3734K kernel code, 277K rwdata, 1212K rodata, 14780K init, 216K bss, 20880K reserved, 0K cma-reserved)
Virtual kernel memory layout:
    vector  : 0xffff0000 - 0xffff1000   (   4 kB)
    fixmap  : 0xffc00000 - 0xfff00000   (3072 kB)
    vmalloc : 0xc4800000 - 0xff800000   ( 944 MB)
    lowmem  : 0xc0000000 - 0xc4000000   (  64 MB)
    modules : 0xbf000000 - 0xc0000000   (  16 MB)
      .text : 0xc0008000 - 0xc04dce24   (4948 kB)
      .init : 0xc04dd000 - 0xc134c000   (14780 kB)
      .data : 0xc134c000 - 0xc13915a0   ( 278 kB)
       .bss : 0xc13915a0 - 0xc13c77d0   ( 217 kB)
----- snip -----
NET: Registered protocol family 1
Kernel panic - not syncing: write error
CPU: 0 PID: 1 Comm: swapper Not tainted 4.4.207 #2
Hardware name: NUC980
Backtrace:
[<c0012ec0>] (dump_backtrace) from [<c00130ac>] (show_stack+0x18/0x1c)
 r6:c392a440 r5:c046f548 r4:c047848c r3:00000000
[<c0013094>] (show_stack) from [<c017e298>] (dump_stack+0x20/0x28)
[<c017e278>] (dump_stack) from [<c0075f28>] (panic+0xb0/0x240)
[<c0075e7c>] (panic) from [<c04dfed4>] (populate_rootfs+0x38/0x250)
 r3:00000001 r2:c3f6e050 r1:c046f548 r0:c047848c
 r7:c04fb828
[<c04dfe9c>] (populate_rootfs) from [<c00095b8>] (do_one_initcall+0x88/0x1ec)
 r10:00000076 r9:00000000 r8:c04dfe9c r7:c04fb828 r6:c392a440 r5:c134fa60
 r4:c134fa60
[<c0009530>] (do_one_initcall) from [<c04dde18>] (kernel_init_freeable+0x108/0x1cc)
 r10:00000076 r9:c04fb838 r8:c0501384 r7:c04fb828 r6:c13915a0 r5:c13915a0
 r4:00000005
[<c04ddd10>] (kernel_init_freeable) from [<c03a7cd0>] (kernel_init+0x10/0xf4)
 r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c03a7cc0
 r4:00000000
[<c03a7cc0>] (kernel_init) from [<c000fdf8>] (ret_from_fork+0x14/0x3c)
 r4:00000000 r3:ffffffff

This is the env.txt used

baudrate=115200
bootdelay=1
stderr=serial
stdin=serial
stdout=serial
setspi=sf probe 0 30000000
loadkernel=sf read 0x7fc0 0x200000 0x1E00000
mtdparts=mtdparts=nand0:0x200000@0x0(u-boot),0x1E00000@0x200000(kernel),-(user)
bootargs=noinitrd ramdisk_size=30720 console=ttyS0,115200n8 oops=panic panic=10 rdinit=/sbin/init mem=64M ubi.mtd=2 mtdparts=nand0:0x200000@0x0(u-boot),0x1E00000@0x200000(kernel),-(user) ignore_loglevel
bootcmd=run setspi;run loadkernel;bootm 0x7fc0

uBoot reports these mtdparts

=> mtdparts

device nand0 <nand0>, # parts = 3
 #: name                size            offset          mask_flags
 0: u-boot              0x00200000      0x00000000      0
 1: kernel              0x01e00000      0x00200000      0
 2: user                0x06000000      0x02000000      0

active partition: nand0,0 - (u-boot) 0x00200000 @ 0x00000000

NAND layout

0x0000000 - u-boot-spl.bin  .5M
0x0080000 - env.txt         .5M
0x0100000 - u-boot.bin      1M
0x0200000 - uImage         30M
0x2000000 - application    96M
0x8000000

/proc/iomem from working/smaller uImage

# cat /proc/iomem
00000000-03ffffff : System RAM
  00008000-004d4df3 : Kernel code
  00eee000-00f67a9b : Kernel data
b0004000-b0004fff : nuc980-gpio.0
b0008000-b0008fff : nuc980-dma
b0009000-b0009fff : nuc980-dma
b0012000-b0012fff : nuc980-emac0
b0015000-b0015fff : nuc980-ehci
  b0015000-b0015fff : ehci_hcd
b0017000-b0017fff : nuc980-ohci.0
  b0017000-b0017fff : ohci_hcd
b0018000-b0018fff : nuc980-sdh
  b0018000-b0018fff : nuc980-sdh
b001c000-b001ffff : nuc980-crypto
  b001c000-b001ffff : nuc980-crypto
b0043000-b0043fff : nuc980-nadc
b0060000-b0060fff : nuc980-qspi0.0
  b0060000-b0060fff : nuc980-qspi0
b0061000-b0061fff : nuc980-spi0.0
  b0061000-b0061fff : nuc980-spi0
b0090000-b0090fff : nuc980-sc.0
  b0090000-b0090fff : nuc980-sc
b0091000-b0091fff : nuc980-sc.1
  b0091000-b0091fff : nuc980-sc

15M Build with BR2_LINUX_KERNEL_LZMA=y, BR2_TARGET_ROOTFS_CPIO_LZMA=y, CONFIG_KERNEL_LZMA=y, CONFIG_RD_LZMA=y, CONFIG_DECOMPRESS_LZMA=y

     2871 Oct  7 22:58 u-boot-spl.bin
   351360 Oct  7 22:58 u-boot.bin
 25330176 Oct  8 13:08 rootfs.cpio
  5420787 Oct  8 13:08 rootfs.cpio.lzma
 12954944 Oct  8 13:09 uImage
 16748928 Oct  8 13:09 Image

In Size constraints of initramfs on ARM? it is mentioned that there is no size constraint.

I tried several things like increasing CONFIG_BLK_DEV_RAM_SIZE in linux and increasing CONFIG_SYS_BOOTM_LEN in uboot, which didn't help.

The BSP i'm working with is based on buildroot 2016.11.1 If more information/context is needed, please let me know. TIA

Kernel config Buildroot config

You need to provide more details such as how is the **cpio** archive loaded for the kernel (e.g. linked with kernel?), and a memory map when the kernel boots (e.g. hand-off from U-Boot?). — sawdust, Oct 02 '21 at 23:57
@sawdust I added some more info: env.txt, uboot mtdparts and the buildroot command that links initramfs into the kernel. Are those enough details? — varta, Oct 03 '21 at 19:12
What kernel version is this? Are you using ATAGs or Device Tree & FIT image? Does the uImage contain a zImage? What is the LOADADDR for your kernel image? If it's also 0x8000 like the contents of the uImage, then that is an inefficient memory footprint for booting. — sawdust, Oct 03 '21 at 21:14
Linux 4.4.207, ATAGS=Y, BR2_LINUX_KERNEL_ZIMAGE is not set, BR2_LINUX_KERNEL_UIMAGE_LOADADDR="" This is uncharted territory for me so I'm not sure at what the LOADADDR for the kernel image is. I will investigate! — varta, Oct 03 '21 at 21:54
*"BR2_..._ZIMAGE is not set"* -- You already stated that you're booting a **uImage**, so I asked what is inside that **uImage**. See https://stackoverflow.com/questions/22322304/image-vs-zimage-vs-uimage Assuming that you do have a **zImage**, then you are loading the **uImage** at a non-optimal location, because even more memory is used when the image is *relocated* before the **zImage** can *decompress*. See https://stackoverflow.com/questions/31725605/building-kernel-uimage-using-loadaddr/31754205#31754205 Have U-Boot load your **uImage** to a (much) higher address instead of 0x7fc0. — sawdust, Oct 04 '21 at 00:35
Tried several higher addresses, like: sf read 0x77fc0 0x200000 0x1E00000; bootm 0x77fc0, but always failing with populate_rootfs. Also the smaller kernel is successfull booting with read 0x1fc0 0x200000 0x1E00000; bootm 0x1fc0 reporting ## Booting kernel from Legacy Image at 00001fc0. Leaving me wondering how... — varta, Oct 05 '21 at 10:00
So you think 0x77fc0 is much higher than 0x7fc0? Apparently you didn't bother to read the links provided. The recommended load address is at least 32MiB from the start of physical memory. Again you need to make a memory map of boot images. You have a useless(?) ramdisk, and need extra memory to relocate the kernel image for decompression. — sawdust, Oct 05 '21 at 19:58
Sorry, your help is really appreciated. But I have a hard time wrapping my head around this stuff. I added a memory map and I tried 0x2007fc0 which also fails. If I go higher U-boot is resetting. — varta, Oct 07 '21 at 08:45
At the moment I have a work around by keeping the kernel smaller. And don't have much time to fix this problem / follow this path. — varta, Oct 07 '21 at 08:55
*"If I go higher U-boot is resetting"* -- You're overwriting the heap of U-Boot. Do you really need to load 0x1e00000 bytes for the **uImage**? What is the actual size of **uImage** when written to NAND? And you can dispense with the 0x7fco offset; it buys you nothing. Your *memory* map looks bogus; why does it look just like the NAND layout? Memory is RAM, not NAND. U-Boot relocates itself to high memory. What *"application"* is there during boot? Why does your memory map go up to *"0x8000000"*? — sawdust, Oct 07 '21 at 09:17
uImage written to NAND is 17M. I tried with "nand read 0x2BB8000 0x200000 0x1200000; bootm 0x2BB8000" still populate_rootfs error. Sorry the map is the NAND layout. I am not sure how to get it for RAM, but I included /proc/iomem — varta, Oct 07 '21 at 10:26
You may be encountering the limits of available physical RAM during boot. Your *compressed* image is already 17MB, more than a quarter of your physical RAM. For a conservative 3:1 expansion, you might need another 51MB, for a total of 68MB. Get the actual size of **Image** from the kernel **arch/arm/boot/** directory. With U-Boot also in RAM, there's not enough memory to accomplish boot. Once the kernel starts, available memory is constrained by the footprint of the **cpio** archive. Try compressing both kernel & rootfs archive using lzma. — sawdust, Oct 07 '21 at 20:22
Ok, I created a new kernel with lzma, Image = 15M. I listed the files above in the question. Rootfs.cpio is uncompressed 24M. Those are sizes I didn't expect to fit in a uImage of 12M. I guess I reached the limit with current RAM. I tried loading with nand read 0x2208000 0x200000 0x1C00000; bootm 0x2208000 resulting in the populate_rootfs error. — varta, Oct 08 '21 at 12:06

Getting a panic at populate_rootfs with bigger initramfs

0 Answers0