3

We are building a custom android board based on an imx6 SoC. the android version used is quite old (KitKat 4.4.2), and so is the kernel (3.0.35). We are dealing with an issue that we haven't figured out yet.

Usually, when everything works fine, the reboot of the board takes 5-6 second top. But sometimes, the reboot of the board takes a long time, ranging anywhere from 1.30 minute up to 2.30 minutes.

What we would like to know is, first, which module / function is the kernel stuck in.

We suspect this could be an eMMC problem, but this is a longshot guess and we really have no clue of what is going on at this point.

Do you guys know of ways to make the kernel extra verbose ? like print every function call ? Could kgdb or similar debugging tools help us at this point ?

Thanks,

Regards,

Vauteck

EDIT: So we made progress in the search of the problem. Turns out the kernel is stuck in the arm_machine_restart() function in arch/arm/kernel/process.c. Specifically, it's stuck after the call to cpu_proc_fin() function, which for our board is defined as cpu_v7_proc_init in arch/arm/mm/proc-v7.S. The code of this function is in assembly : 

mrc p15, 0, r0, c1, c0, 0       @ ctrl register
bic r0, r0, #0x1000         @ ...i............
bic r0, r0, #0x0006         @ .............ca.
mcr p15, 0, r0, c1, c0, 0       @ disable caches
mov pc, lr

We are not the only ones that encountered this issue. (thread on NXP forum here) We tried commenting out the line

// bic  r0, r0, #0x0006         @ .............ca.

Now the function never blocks but sometimes the board still doesn't reboot immediately. We are still looking for insights and suggestions at this point. Thanks for reading guys.

Vauteck
  • 31
  • 2

1 Answers1

3

If you enable CONFIG_PRINTK_TIME in the kernel, dmesg will print the time before the logs (in seconds). This enables you to search for time gaps between lines and maybe you're able to find what is causing this problem.

If you've found out that the problem indeed exists in the kernel, it's likely that you can enable some CONFIG_DEBUG_* configuration item or define CONFIG_DEBUG in the driver to obtain more information. Otherwise, printk will be the best you've got.

Also, take a look the the following kernel configurations:

CONFIG_DEBUG_LL
CONFIG_DEBUG_IMX_UART
CONFIG_DEBUG_IMX6Q_UART
CONFIG_EARLY_PRINTK
CONFIG_EARLY_PRINTK_DIRECT

To be complete: You can make use of logcat to see whether or not some initialisation delays the boot. If your company builds the hardware, I think it pays off to see what the chip is doing with a scope (because I don't immediately think that Linux is delaying the boot), but not before you know for certain that multiple boards have the same problem.

I'm interested in what you will find. Keep me (us) updated ;-)

Bayou
  • 3,293
  • 1
  • 9
  • 22
  • Thank you for your answer and suggestions. We will check the kernel options you mentionned to see if we can output more logs. I should have been more precised in my original post, but this is the shutdown of the board that takes a long time, not the actual boot after the reboot. I'll keep you updated on our finding :-). – Vauteck Sep 30 '19 at 06:52
  • 1
    Connect the uart and enable the printk time. The kernel logs are printed to uart, thus you will be able to see what's taking too long to shutdown just as you would with the startup :-) – Bayou Sep 30 '19 at 08:26