1

I am booting Android on an IMX53 Sabre tablet. The last few lines seen on serial port as android boots up normally is as follows:

warning: `rild' uses 32-bit capabilities (legacy support in use)
pmem: request for physical address of pmem region from process 2262.
request_suspend_state: on (3->0) at 12032459753 (2000-01-03      
01:08:28.336600001 U
TC)
Unhandled fault: external abort on non-linefetch (0x1018) at 0x40a85054
Unhandled fault: external abort on non-linefetch (0x1018) at 0x40a85054

Note, these external aborts do not stop the boot process and Android boots normally. Now I want to use these external aborts to test my monitor abort handler code. I want the ability to trap external aborts to monitor.

In initialization of secure monitor, I do the following:

mrc p15, 0, r4, c1, c1, 0
bic r4, #0x66                      @ clear AW, IRQ, FIQ
orr r4, #0x19                      @ set FW, EA, NS
//orr r4, #0x01                     @ previously, just set NS
mcr p15, 0, r4, c1, c1, 0

My monitor vector table looks as follows:

.global tz_monitor
.align 5
tz_monitor:
@ Monitor
   b    _monitor_reset      @ Reset
   b    _monitor_undef      @ Undef
   b    smc_handler         @ SMC
   b    _monitor_prefetch   @ Prefetch
   b    _monitor_da         @ Data abort
   nop                      @ RESERVED
   b    _monitor_irq        @IRQ
   b    _monitor_fiq        @FIQ

An example exception handler function just prints a statement and looks as follows: In assembly:

.global _monitor_prefetch
_monitor_prefetch:
    push {lr}
    bl monitor_prefetch
    pop {lr}
    movs pc, lr

In C:

void monitor_prefetch(void) {
    printf("In Monitor's Prefetch Handler\n");    
}

I add this vector table in initialization of secure monitor as follows:

ldr r0, =tz_monitor       @ Get address of Monitors vector table
mcr p15, 0, r0, c12, c0, 1 @ Write Monitor Vector Address Register

There is change with these modifications, namely kernel crashes at the point of external abort. But I do not get the print statement in the monitor handler. I get the following output on serial now, when booting Android:

warning: `rild' uses 32-bit capabilities (legacy support in use)
pmem: request for physical address of pmem region from process 2262. 
request_suspend_state: on (3->0) at 12005333628 (2000-01-03   
00:38:52.322241626 UTC)
Bad mode in prefetch abort handler detected
Internal error: Oops - bad mode: 0 [#1] PREEMPT
last sysfs file: /sys/devices/platform/pwm-backlight.0/backlight/pwm-  
backlight.0/brightness
Modules linked in:
CPU: 0    Not tainted  (2.6.35.3-01265-g8f56f17 #6)
PC is at 0x77802570
LR is at 0xafd0bf6c
pc : [<77802570>]    lr : [<afd0bf6c>]    psr: 200001d6
sp : ce0e7fb0  ip : 8151c780  fp : 00000001
r10: 00000000  r9 : 40a85000  r8 : 00000002
r7 : 4090ae64  r6 : 8151c890  r5 : 00000018  r4 : 40207000
r3 : 40a85000  r2 : 00000001  r1 : 60000010  r0 : 00000000
Flags: nzCv  IRQs off  FIQs off  Mode UK6_32  ISA ARM  Segment user
Control: 10c5387d  Table: 83668019  DAC: 00000015
Process Binder Thread # (pid: 2307, stack limit = 0xce0e62e8)
Stack: (0xce0e7fb0 to 0xce0e8000)
7fa0:                                     00000000 60000010 00000001    
40a85000
7fc0: 40207000 00000018 8151c890 4090ae64 00000002 40a85000 00000000  
00000001
7fe0: 8151c780 ce0e7fb0 afd0bf6c 77802570 200001d6 ffffffff ffc75a9a 
dfd4eed7
Code: bad PC value
---[ end trace d8447dd37d1d45d8 ]---
Kernel panic - not syncing: Fatal exception
[<c003e58c>] (unwind_backtrace+0x0/0xf0) from [<c04862f0>]     
(panic+0x6c/0xe0)
[<c04862f0>] (panic+0x6c/0xe0) from [<c003d420>] (die+0x2b4/0x304)
[<c003d420>] (die+0x2b4/0x304) from [<c003d4ac>] (bad_mode+0x3c/0x5c)
[<c003d4ac>] (bad_mode+0x3c/0x5c) from [<afd0bf6c>] (0xafd0bf6c)

Do I need some other setting to trap the abort to monitor mode? Eventually I will set CSU to make peripherals like I2C0/IPU secure and trap and emulate aborts for those peripherals. But I need to do this first step of directing an abort to a monitor handler function. Can someone please help?

EDIT

First let me try the prefetch abort handler without any printf (to reduce where errors are coming from). So the goal is: abort comes to the monitor, and the monitor just gives back control to the NW again, as this abort is not of interest.

Essentially, there is only one bit in SCR to specify what should happen to any abort. But I might be interested only in data aborts (that also when access happens for some particular peripheral). So this first step is to just give back control to NW, when the prefetch abort happens.

The following code as prefetch handler has the same output: crash with "Bad mode in prefetch abort handler detected".

.global tz_monitor
.align 5
tz_monitor:
    @ Monitor
    nop @ Reset      - not used by Monitor
    nop @ Undef      - not used by Monitor
    b   smc_handler
    b   pabort_handler @ Prefetch   - can by used by Monitor
    b   dabort_handler  @ Data abort - can by used by Monitor
    nop @ RESERVED
    nop @ IRQ        - can by used by Monitor
    nop @ FIQ        - can by used by Monitor

.global pabort_handler
pabort_handler:
    sub r14, r14, #4            @ on excpetion, processor stores     current pc to r14 or link register (lr), decrement by 4 for prefetch abort
    stmfd sp!, {r14}            @ save lr, to know where to return from exception
    mrs r14, spsr               @ on exception, processor stores current cpsr to spsr, read this to r14
    stmfd sp!, {r0-r12, r14}        @ save r14 and therefore spsr, also store general purpose registers r0-r12

    mrc p15, 0, r4, c1, c1, 0           @ Read Secure Configuration Register data
    bic r4, #NS_BIT                     @ clear NS bit
    mcr p15, 0, r4, c1, c1, 0           @ Write Secure Configuration Register data

    //bl monitor_prefetch @no printing for now

    mrc p15, 0, r4, c1, c1, 0           @ Read Secure Configuration Register data
    orr r4, #NS_BIT                     @ Set NS bit
    mcr p15, 0, r4, c1, c1, 0           @ Write Secure Configuration Register data

    ldmfd sp!,{r0-r12, r14}             @ restore from stack general purpose registers and read spsr value to r14 
    msr spsr, r14               @ restore spsr from r14
    ldmfd sp!, {pc}^            @ restore from stack lr-4 and set pc, the ^ denotes cpsr to be set with spsr

.global dabort_handler
dabort_handler:
    sub r14, r14, #8            @ on excpetion, processor stores current pc to r14 or link register (lr), decrement by 8 for data abort
    stmfd sp!, {r14}            @ save lr, to know where to return from exception
    mrs r14, spsr               @ on exception, processor stores current cpsr to spsr, read this to r14
    stmfd sp!, {r0-r12, r14}        @ save r14 and therefore spsr, also store general purpose registers r0-r12

    mrc p15, 0, r4, c1, c1, 0           @ Read Secure Configuration Register data
    bic r4, #NS_BIT                     @ clear NS bit
    mcr p15, 0, r4, c1, c1, 0           @ Write Secure Configuration Register data

    //bl monitor_prefetch  @no printing for now

    mrc p15, 0, r4, c1, c1, 0           @ Read Secure Configuration Register data
    orr r4, #NS_BIT                     @ Set NS bit
    mcr p15, 0, r4, c1, c1, 0           @ Write Secure Configuration Register data

    ldmfd sp!,{r0-r12, r14}             @ restore from stack general purpose registers and read spsr value to r14 
    msr spsr, r14               @ restore spsr from r14
    ldmfd sp!, {pc}^            @ restore from stack lr-4 and set pc, the ^ denotes cpsr to be set with spsr
rijurekha
  • 21
  • 3
  • Okay, as I said your `_monitor_prefetch` is not saving enough registers. You need to save all registers; they are not banked between secure/normal worlds. You will randomly corrupt Linux registers when/where the 'external abort' occurs. You are sharing resources with Linux. You haven't specified how `printf` works? Use a polled mode UART to send characters and make sure you 'flush' from the monitor. Be aware that I/O configuration **AND** clocks are also shared with Linux; How will that affect the UART? Also, you need to switch back to 'NS' when you return to Linux. – artless noise Mar 30 '17 at 12:46
  • Thanks @artlessnoise. I edited the question. Regarding what I have in secure world: I have these files https://www.dropbox.com/sh/t98sceuw6hp885g/AABZ1EK-7Wwngc8Y5od88fr2a?dl=0 added/modified in lib_arm/ folder of uboot-imx. I only change the uboot.bin when I compile this. The linux kernel and Android images I do not change. U-boot has other printf-s, but I am not clear how clock/UART is shared across worlds, if at all. Should I try something like https://github.com/finallyjustice/imx53qsb-code/tree/master/serial? I will save/restore registers in the handler and toggle from NS and back. – rijurekha Mar 30 '17 at 15:05
  • Okay, Where is your 'booting linux' message from U-boot? On the same serial as Linux console? They will conflict. Linux will set the iomux(pinctrl) and the clock tree which the 'setbrg()' jazz depends on for a valid baud rate. When I did this, I used a seperate serial port for TZ (with no Linux DT reference to the serial port) and wrote assembler polling routines (like the early console debug in Linux). Just pass fixed strings (no formatting like `printf`) and you can have a `printhex()`... This minimizes registers and memory to succeed in printing, minimizing things to get right. – artless noise Mar 30 '17 at 16:05
  • The message *Bad mode in prefetch abort handler detected* means you didn't save the `spsr` on monitor entry and restore this upon return to the normal world; so the `movs pc,lr` seems like the spsr changed during the `printf` (perhaps due to yet another exception). Normally you save all registers (including spsr) in a 'world context' and restore them on return. – artless noise Mar 30 '17 at 16:15
  • Why are you using `R2`? That is also shared with Linux. You need to set **SP_monitor**. So before booting Linux, you need to allocate memory and assign it to `R13` while in monitor mode. This is then the monitor stack. Change the `R2` to either `R13` or `SP`. – artless noise Apr 03 '17 at 13:13
  • @artlessnoise, I changed code in the **EDIT** above. I have posted a separate question at [link](http://stackoverflow.com/questions/43237171/imx53-initialize-stacks‌)​, as `sp` might not be set correctly. It can also be a "mode" issue. Are we in "abort" mode when we enter this handler, or in "monitor" mode? According to [link](http://www.iti.uni-stuttgart.de/~radetzki/Seminar06/08_report.pdf), Table 2.4, the mode change on exception is done by processor. Copying back the spsr restores the mode where the exception occurred. Do we need to change the processor mode anywhere within the handler? – rijurekha Apr 06 '17 at 07:57
  • According to [link](http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0344h/Babeeidg.html), with `bic r4, #0x66 orr r4, #0x19`, the EA will branch to monitor mode, and not abort mode. So when we enter the handler, we are in `MON` mode. When we write back spsr, we are back in `USR` mode? The error is still **Bad mode in prefetch abort handler detected**, so where should we change processor mode to what? – rijurekha Apr 06 '17 at 08:10
  • Move the `p15` stuff to after the register saving. You will still nuke `r4` form whatever Linux was doing. You can try `SUBS PC,R14_abt,#4` as a minimal to see and then add things. – artless noise Apr 06 '17 at 13:04
  • @artlessnoise, sorry about the `p15` and nuking `r4`. Changed in EDIT above. Added dabort_handler with `#8`. The operations are completely symmetric now: save registers, clear NS, set NS, restore registers -- not sure what functionality this is achieving. The output remains the same. Also same output with only `msr cpsr_c, #Mode_ABT | I_Bit | F_Bit subs pc, r14, #4` in pabort_handler and #8 equivalent in dabort_handler. I get compiler error with `r14_abt`, so first changed mode to `abt` and then did `subs` with `r14`. Maybe the issue is not with the handler at all, but somewhere else? – rijurekha Apr 06 '17 at 13:38
  • Sorry, I meant just `SUBS PC,R14,#4` in `pabort_handler`. It seems weird that Linux is reporting a mode of '0'. Do the addresses 0x77802570 and 0xafd0bf6c make sense? They seem like garbage. – artless noise Apr 06 '17 at 14:04
  • Same output with `SUBS PC,R14,#4`. I don't know whether the addresses make sense or not, how can I tell? What does mode 0 mean among the 8 ARM processor modes? – rijurekha Apr 06 '17 at 14:36
  • I don't know if this is important. But the crash shows different values for `last sysfs file: /sys/devices/platform/max17085_bat.0/power_supply/battery/techn ology` or `last sysfs file: /sys/devices/virtual/usb_composite/usb_mass_storage/enable` or `last sysfs file: /sys/devices/platform/pwm-backlight.0/backlight/pwm- backlight.0/brightness`, though the `PC` and `LR` remains the same as `0x77802570` and `0xafd0bf6c`. – rijurekha Apr 06 '17 at 14:50
  • The mode '0' sounds familiar. You need to enable 'cache' with an 'smc' somehow. See: [cache-l2x0.c](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arm/mm/cache-l2x0.c?id=refs/tags/v4.11-rc5#n871). Also, are you sure your device tree made it to the normal boot? The mode '0' was some non-related issue when I was getting Linux to boot in the normal world. Ie, I remember it being a side-effect of something else. Also, you need to initialize any shadowed CP15 registers as they are garbage in the Normal world after reset; whereas secure world have known values. – artless noise Apr 07 '17 at 13:17
  • Do you mean the device tree made to the normal boot, when I started normal boot of linux kernel +Android, or after the crash? Now I have `bic r4, #0x66 orr r4, 0x19` for cp15 to trap EAs to the monitor mode. The original `orr r4, #0x01` normally booted Android (with those EA's printed on console, which are now causing the kernel to crash). I could use the camera and take photos on the Sabre once the Android booted. Doesn't that need the device tree to work with the normal boot? – rijurekha Apr 07 '17 at 16:01
  • You mean the `CP15` registers are getting bad at the EA, and I need to set their values in the handler? How do I know what values they had in the normal world at the point of `EA`, does the processor save this before switching to the monitor handler? Without trapping the `EA` to monitor, the normal world was booting Android, so I think the `CP15` registers had correct values, even though I don't initialize them in reset? – rijurekha Apr 07 '17 at 16:06
  • If your Linux boots fine in 'normal mode' with some SCR settings (CP15), then neither the device tree nor the CP15 initialization is an issue. Some versions of Linux rely on default/reset values of the shadowed CP15 registers. They are garbage/random when booting in normal mode. This question, [Enabling EA on an ARM CPU](http://stackoverflow.com/questions/23962450/enabling-external-aborts-on-an-arm-cpu), was actually motivated by an EA issue with Trustzone. It could also be where I saw 'mode 0'. I had to enable the 'A' bit before any Linux code ran to take an EA and then boot Linux. – artless noise Apr 09 '17 at 15:05
  • I tried to enable the `A` bit in `cpsr` just before setting cp15 with `bic r4, #0x66 orr r4, 0x19` for normal world. `mrs r0, cpsr orr r0, r0, #0x00000200 msr cpsr, r0` doesn't help and has the same crash. My abort handlers have only `subs pc, r14, #4` now, not using any stack. – rijurekha Apr 21 '17 at 11:16

0 Answers0