how does this assembly code get the difference between physical offset and page offset?

Question

While studying the linux source code, I see a lot of asssembly code that looks like this:

adr r0, 1f
ldmia   r0, {r3-r7}
mvn ip, #0
subs    r3, r0, r3  @ PHYS_OFFSET - PAGE_OFFSET
...
    .align
1:  .long   .
    .long   __pv_table_begin
    .long   __pv_table_end
2:  .long   __pv_phys_pfn_offset
    .long   __pv_offset

first, it starts with adr and in the first line above, I understood that adr r0, 1f means that it will save the address where 1: starts in r0.

ldmia r0, {r3-r7} means that it will load values starting from the address saved in r0(which points to 1:) to registers r3,r4,r5,r6,r7. Therefore,

r3=.

r4=__pv_table_begin

r5=__pv_table_end

r6=__pv_phys_pfn_offset

r7=__pv_offset.

Now, the part part that I don't get: subs r3,r0,r3 I'm not entirely sure what r3=. means but I guessed that r3 will eventually contain a value of its own address.

Vaguely speaking, r0 and r3 values are both address values pointing to the same position 1:. But I guess their values are different because one if physical address and the other is a virtual address. (<- this is purely my guess).

That is why I thought somehow the code is trying to get the difference between these two through subs r3, r0, r3.

I'm not sure if my guess if correct, and even so, I do not know which one is the physical address and the virtual address. Also, the comment mentions that the subtraction will produce the difference between the physical offset and page offset. I've read about the pages related to virtualization of memory, but I cannot related this knowledge to this offset subtraction.

I'm not an expert of ARM but the `adr` computes the address of `1` at runtime (i.e. whenever that code has been relocated) while the gas `.` directive is the offset w.r.t. the current section (or similar, I'm not expert of GAS either). — Margaret Bloom, Feb 20 '17 at 11:27

old_timer · Accepted Answer · 2017-02-20T13:09:02.787

Why dont you just try it?

hello:
    .long .
    .long 0x11111111
    .long 0x22222222
    .long 0x33333333
    .long 0x44444444
    .long 0x55555555
    .long 0x66666666

.globl TEST
TEST:
    adr r0,hello
    bx lr

linked and disassembled

0000803c <hello>:
    803c:   0000803c    andeq   r8, r0, r12, lsr r0
    8040:   11111111    tstne   r1, r1, lsl r1
    8044:   22222222    eorcs   r2, r2, #536870914  ; 0x20000002
    8048:   33333333    teqcc   r3, #-872415232 ; 0xcc000000
    804c:   44444444    strbmi  r4, [r4], #-1092    ; 0xfffffbbc
    8050:   55555555    ldrbpl  r5, [r5, #-1365]    ; 0xfffffaab
    8054:   66666666    strbtvs r6, [r6], -r6, ror #12

00008058 <TEST>:
    8058:   e24f0024    sub r0, pc, #36 ; 0x24
    805c:   e12fff1e    bx  lr

TEST returns 0x803C as we both expected.

The first item in the list though may be your mystery. Notice how they use the dot shortcut to indicate here or this address so the first item in the list is the address of the beginning of the list. which r0 had already they could have just done a mov r3,r0 but maybe burning that instruction vs just loading it and burning ram with one instruction. who knows...

so

.globl TEST
TEST:
    adr r0,hello
    ldmia r0,{r3}

    mov r3,r0
    bx lr

which returns the same 0x803C value.

Now

.globl TEST
TEST:
    adr r0,hello
    ldmia r0,{r3}
    subs r3,r0,r3

    mov r0,r3
    bx lr

and as expected that returns zero, so what is the point of all of this? Note that this whole section is position independent right? Well what if I change my linker to think this is being loaded somewhere else...

MEMORY
{
    ram : ORIGIN = 0xA000, LENGTH = 0x1000000
}

producing

0000a03c <hello>:
    a03c:   0000a03c    andeq   r10, r0, r12, lsr r0
    a040:   11111111    tstne   r1, r1, lsl r1
    a044:   22222222    eorcs   r2, r2, #536870914  ; 0x20000002
    a048:   33333333    teqcc   r3, #-872415232 ; 0xcc000000
    a04c:   44444444    strbmi  r4, [r4], #-1092    ; 0xfffffbbc
    a050:   55555555    ldrbpl  r5, [r5, #-1365]    ; 0xfffffaab
    a054:   66666666    strbtvs r6, [r6], -r6, ror #12

0000a058 <TEST>:
    a058:   e24f0024    sub r0, pc, #36 ; 0x24
    a05c:   e8900008    ldm r0, {r3}
    a060:   e0503003    subs    r3, r0, r3
    a064:   e1a00003    mov r0, r3
    a068:   e12fff1e    bx  lr

but still execute at the same place gives 0xFFFFE000 which is -0x2000 because of the direction I changed my linker if I change it to 0x5000 instead of 0xA000 I get 0x3000 as the difference.

So what this code is doing is

.long .

is compile time the adr is runtime and uses the runtime pc so this code is detecting the difference between the actual memory address where the table is and the compile time address where the table is. if the items in the table are compile time addresses

hello:
    .long .
    .long one
    .long two
    .long three
one:
    .long 0x44444444
two:
    .long 0x55555555
three:
    .long 0x66666666


0000503c <hello>:
    503c:   0000503c    andeq   r5, r0, r12, lsr r0
    5040:   0000504c    andeq   r5, r0, r12, asr #32
    5044:   00005050    andeq   r5, r0, r0, asr r0
    5048:   00005054    andeq   r5, r0, r4, asr r0
0000504c <one>:
    504c:   44444444    strbmi  r4, [r4], #-1092    ; 0xfffffbbc
00005050 <two>:
    5050:   55555555    ldrbpl  r5, [r5, #-1365]    ; 0xfffffaab
00005054 <three>:
    5054:   66666666    strbtvs r6, [r6], -r6, ror #12

Then in order to use this jump table or look up table you need to know the compiled address vs the runtime address so you can adjust the compile time addresses in the code.

Using terms like physical and page, I think page is wrong, but it could be virtual vs link time (I guess compiled is the wrong term too, link time vs runtime) it is still runtime vs link time whether the reason for the difference is position independence or virtualization. If running on an operating system the link time and runtime should be the same the physical cannot be detected this way as the processor (adr) at least as documented sees the PC based value and the PC doesnt know physical from virtual, that is off the edge of the core in the mmu. So I think both terms physical and page are incorrectly used here, but that is just my opinion.

If you remove -fPIC from your compiler options and not make it position independent code I wonder if it would not bother with all this and just use the table as is.

ahh, you said the linux source code, this is not compiled code. they are making it position independent. If I remember right you can load the kernel and at least the first part that uncompresses the real kernel and drivers is position independent. Perhaps the whole thing is (what a performance hit). — old_timer, Feb 20 '17 at 13:11
perhaps the author is used to x86 where a term like page does make sense. this could be cut and pasted then translated to arm without changing the comments. — old_timer, Feb 20 '17 at 13:13
.long vs .word is another clue there, a word in x86 is 16 bits in arm 32 bits a long varies in x86 (16 bit days the C version of long was 32 bits, in the 32 bit days it was 32 bits, but with 64 bit it changed to 64 at least with one/some compilers). Although this is all gnu assembler so that in and of itself doesnt mean anything due to the nature of gnu assembler habits vs compilers or reality. In this case .word and .long are interchangeable, I find it odd that .long was chosen here, when an arm programmer would probably think word not long. — old_timer, Feb 20 '17 at 13:17

score 1 · Answer 2 · edited May 23 '17 at 11:46

There are some concepts you need to understand in the code.

Code gets a 'link' address where absolute addressing is resolved.
The run (PC) address maybe different.

The code you are looking at is relocation code which will fix up 'link' absolute addressing with the run time PC address.

adr r0, 1f
ldmia   r0, {r3-r7}
mvn ip, #0
subs    r3, r0, r3  @ PHYS_OFFSET - PAGE_OFFSET
...
    .align
1:  .long   .
    .long   __pv_table_begin
    .long   __pv_table_end
2:  .long   __pv_phys_pfn_offset
    .long   __pv_offset

The adr r0, 1f and 1: .long . seem the same. However, there is a subtle difference. The 1: .long . line will store a link address as the '1:' local label. The adr r0, 1f will convert to add r0, pc, #offset and so a runtime address will be placed in R0. The ldmia r0, {r3-r7} loads many values, but the R3 value is the link address of the local label. Finally, subs r3, r0, r3 will put a difference between the run address and the link address in R3; a correction term.

The table is then a list of link addresses which need to have a fix-up applied. This allows 'non-PIC' code to run at different addresses.

The comment right above seems to be helpful,

/* __fixup_pv_table - patch the stub instructions with the delta between
 * PHYS_OFFSET and PAGE_OFFSET, which is assumed to be 16MiB aligned and
 * can be expressed by an immediate shifter operand. The stub instruction
 * has a form of '(add|sub) rd, rn, #imm'.
 */

This depends on the Kconfig value ARM_PATCH_PHYS_VIRT which seems to be needed for the Keystone2 CPU/SOC as per early_paging_init with machine fix-ups for physical addresses.

The reason this table is needed is only for kernel code which need a physical address of memory, usually to communicate with DMA devices but also quite extensively by the mm (or virtual memory management) code which is performance critical to a paging OS. In Linux an inline function just subtracts/adds an offset of a difference between phys/virt kernel addresses; virt_to_phys, phys_to_virt, etc. When this feature is used in a driver/module it needs to be fixed when the compiled phys/virtual difference is not the same as what happens when the image is run. There is always a contiguous map of kernel memory (virtual remapping is a fixed offset only for kernel addresses)

Some machinery is located in memory.h and maybe instructive in understanding what is going on. Note the comment,

/*
 * Physical vs virtual RAM address space conversion.  These are
 * private definitions which should NOT be used outside memory.h
 * files.  Use virt_to_phys/phys_to_virt/__pa/__va instead.
 *
 * PFNs are used to describe any physical page; this means
 * PFN 0 == physical address 0.
 */

See:

Getting a label to a register
Relocation in ARM assembler
Assemblers and Loader, by David Solomon
GOLD blog by Ian Lance Taylor on a new GNU Linker and linker issues.

Both physical and virtual must be patched; build virtual to run virtual and build physical (PHYS_OFFSET) to run physical. Linux traditionally executed with a 3G/1G split so run virtual was 0xc0000000; however people needed this to change for different reasons. Also the physical memory is different for different SOC/manufacturer. To run a single binary on multiple SOC vendor types, the 'PHYS_OFFSET' is dynamic (well, it is self-modifying code in pv_table jazz above which is patching). What ever the case, the kernel virtual to physical difference is run constant for all addresses. — artless noise, Feb 21 '17 at 13:50

how does this assembly code get the difference between physical offset and page offset?

2 Answers2