18

I'm investigating the BIOS code in my machine (x86_64 Linux, IvyBridge). I use the following procedure to dump the BIOS code:

$ sudo cat /proc/iomem | grep ROM
  000f0000-000fffff : System ROM
$ sudo dd if=/dev/mem of=bios.dump bs=1M count=1

Then I use radare2 to read and disassemble the binary dump:

$ r2 -b 16 bios.dump 
[0000:0000]> s 0xffff0
[f000:fff0]> pd 3
        :   f000:fff0      0f09           wbinvd
        `=< f000:fff2      e927f5         jmp 0xff51c
            f000:fff5      0000           add byte [bx + si], al

I know x86 processor initialization always starts with a 16-bit 8086 environment, and the first instruction to be executed is at f000:fff0, i.e. 0xffff0. So I go to that location and disassemble the code.

To my surprise, the first instruction is WBINVD, whose functionality is to invalidate the cache, which seems to be irrelevant when the processor is powered on or reset. I would expect the first instruction to be simply a jmp to a lower memory address.

Why is there a WBINVD before jmp?

I've already searched the relevant portion of the Intel manuals, Volume 3 Chapter 9 Processor Management and Initialization, but it doesn't mention anything about WBINVD. I also searched some online resources but didn't find any explanation.

Edit for more info:

After following the jmp instruction to 0xff51c, the code is more interesting; it's doing a self-check:

[f000:f51c]> pd
            f000:f51c      dbe3           fninit
            f000:f51e      0f6ec0         movd mm0, eax
            f000:f521      6631c0         xor eax, eax
            f000:f524      8ec0           mov es, ax
            f000:f526      8cc8           mov ax, cs
            f000:f528      8ed8           mov ds, ax
            f000:f52a      b800f0         mov ax, 0xf000
            f000:f52d      8ec0           mov es, ax
            f000:f52f      6726a0f0ff00.  mov al, byte es:[0xfff0]     ; [0xfff0:1]=0
            f000:f536      3cea           cmp al, 0xea
        ,=< f000:f538      750f           jne 0xff549
        |   f000:f53a      b91b00         mov cx, 0x1b
        |   f000:f53d      0f32           rdmsr  ; check BSP (Boot Strap Processor) flag, if set, loop back to 0xffff0; otherwise, infinite hlt
        |   f000:f53f      f6c401         test ah, 1
       ,==< f000:f542      7441           je 0xff585
      ,===< f000:f544      eaf0ff00f0     ljmp 0xf000:0xfff0
      ||`-> f000:f549      b001           mov al, 1
      ||    f000:f54b      e680           out 0x80, al
      ||    f000:f54d      66be8cfdffff   mov esi, 0xfffffd8c          ; 4294966668
      ||    f000:f553      662e0f0114     lgdt cs:[si]
      ||    f000:f558      0f20c0         mov eax, cr0
      ||    f000:f55b      6683c803       or eax, 3
      ||    f000:f55f      0f22c0         mov cr0, eax
      ||    f000:f562      0f20e0         mov eax, cr4
      ||    f000:f565      660d00060000   or eax, 0x600
      ||    f000:f56b      0f22e0         mov cr4, eax
      ||    f000:f56e      b81800         mov ax, 0x18
      ||    f000:f571      8ed8           mov ds, ax
      ||    f000:f573      8ec0           mov es, ax
      ||    f000:f575      8ee0           mov fs, ax
      ||    f000:f577      8ee8           mov gs, ax
      ||    f000:f579      8ed0           mov ss, ax
      ||    f000:f57b      66be92fdffff   mov esi, 0xfffffd92          ; 4294966674
      ||    f000:f581      662eff2c       ljmp cs:[si]
      |`.-> f000:f585      fa             cli
      | :   f000:f586      f4             hlt
      | `=< f000:f587      ebfc           jmp 0xff585

To conclude the weirdness, this BIOS code is reading itself at 0xffff0 and comparing the byte with 0xea, which is exactly the opcode of a far jump:

            f000:f52a      b800f0         mov ax, 0xf000
            f000:f52d      8ec0           mov es, ax
            f000:f52f      6726a0f0ff00.  mov al, byte es:[0xfff0]     ; [0xfff0:1]=0
            f000:f536      3cea           cmp al, 0xea

If it finds the code at 0xffff0 is a far jump, then it will go into an infinite loop.

More precisely, the APs (Application Processors) will loop infinitely at the hlt instruction, while the BSP (Boot Strap Processor) will loop back to the beginning 0xffff0. Since the code at 0xffff0 won't be changed, we can conclude the BSP will always find the byte being 0xea and will never go out of the loop.

So what's the purpose of this self-checking? I can hardly believe it's a naive attempt to prevent modification.

Boann
  • 48,794
  • 16
  • 117
  • 146
user10865622
  • 455
  • 3
  • 11
  • Maybe related: x86 firmware often puts the CPU into cache-as-ram mode (no-fill mode) in early bootup, which you leave using INVD (like WBINVD but without the write-back first). [What use is the INVD instruction?](https://stackoverflow.com/q/41775371). I don't know if that has anything to do with WBINVD here; I don't see a connection. Maybe they're using it as a pause / delay to give devices on the mobo more time to initialize? It's so slow (and not interruptible) that it's a privileged instruction. If it flushes i-cache, I wonder if it results in the `jmp` instruction being re-fetched? – Peter Cordes Jan 20 '19 at 07:05
  • 3
    Note that what you see at `0xffff0` could be very little related to the real boot strap code. Execution starts with `cs:ip = 0xf000:0xfff0` but the `cs` nevertheless uses `0xffff0000` as a base, resulting in a physical address of `0xfffffff0`. The PCH aliases the legacy ROM hole to this upper range until the BIOS itself initialises it properly. Usually the code in the lower range is the same as the one in the upper range but you should look near the 4GiB to be sure. Finally, remember that legacy BIOS introduced the warm boot, so you may be at `0xffff0` not necessarily only after a boot. – Margaret Bloom Jan 20 '19 at 10:25
  • @MargaretBloom: Ah, I bet the warm boot would explain it. You'd presumably want to make sure all cached writes were committed before entering cache-as-RAM mode, maybe in case of cacheable I/O regions, or just for good measure. I wondered about this not actually being the BIOS entry point because of the `0xffff0000` thing, but with the OP's update this looks very much like what you'd expect. (Saving EAX into MM0 had me suspicious, but then I realized it wasn't XMM0, so it doesn't need the SSE-enable bits in control registers to be set for it to work.) Unless there's similar code elsewhere? – Peter Cordes Jan 20 '19 at 11:48
  • 1
    @PeterCordes I've checked the `0xffff0000` thing, it's the same, so I believe this is the boot code. I'm now more confused by the self-checking thing in the code, as you can see from my update. Do you have any thought about the self-check? I really can't believe it's a silly attempt to prevent code modification. – user10865622 Jan 20 '19 at 12:26
  • I don't know. Some random unlikely ideas: Maybe it's a check against data corruption somehow? I thought maybe if this code can be copied somewhere and re-executed, but then it's checking a full absolute linear address, not relative to CS. But maybe if the original physical memory is remapped at that point? – Peter Cordes Jan 20 '19 at 12:40
  • 2
    BIOS code is typically copied into RAM, maybe the copy changes the `WBINVD` into a far jump so the second time around the boot processor follows that while the others will wait in the `HLT` until the BSP initializes them. – Jester Jan 20 '19 at 14:36
  • 2
    @Jester: Maybe it's something like "BSP loops until some kind of security processor (e.g. Intel Management Engine) changes the `JMP` into a `WBINVD`". – Brendan Jan 20 '19 at 15:20
  • According to slide 14 of [BIOS and System Management Mode Internals](https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=2&ved=2ahUKEwibwZ6Rof_fAhUIinAKHczpCa8QFjABegQIBBAC&url=http%3A%2F%2Fopensecuritytraining.info%2FIntroBIOS_files%2FDay2_04_Advanced%2520x86%2520-%2520BIOS%2520and%2520SMM%2520Internals%2520-%2520UEFI.pptx&usg=AOvVaw399-I85reOjVrE_i2ovPDe), the wbinv instruction was there in UDK2010 but then got later removed in UDK2012. Perhaps it's security-related. I don't know exact what. – Hadi Brais Jan 21 '19 at 16:56

4 Answers4

7

Albeit hard to reason about, remember that the load mov al, byte es:[0xfff0] is not reading from the the BIOS first instruction, even though es is set to 0xf000.

The first instruction is read from 0xfffffff0, the PCH will also probably alias 0xf0000-0xfffff to 0xffff0000-0xffffffff at reset, so when the BSP is booted it will execute the code you dumped.
IIRC, the APs don't boot unless explicitly waken up.

The BSP will then will proceed with initialising the HW (judging from the dump).
At some point it will set the attribute map for the 0xf0000-0xfffff to steer reads and writes (or just writes and then reads) to memory.
The end result is that when a processor (an HW thread) boots it will execute the code from the flash until it perform a far jump.
At the point the cs base is correctly computed as per real-mode rules (pretty much like the unreal mode) and the instruction will be fetched from the 0xf0000-0xfffff (i.e. from the RAM).
All of this while the cs segment value didn't actually change.

The BSP at some point will start its multiprocessor initialisation routine, where it broadcasts to everyone (including himself) an INIT-SIPI-SIPI that will result in a sleep for the APs and a ljmp 0xf000:0xfff0 for the BSP.
The trick here is that the target of the jump, 0xf000:0xfff0, is not the same bus address of the wbinvd instruction.
There could be something else there, probably another initialisation routine.

At the end of the initialisation the BIOS could simply reset the attributes of the 0xf0000-0xfffff to fall through to the flash (so a software reset is possible), preventing (not intentionally) a dump of the intermediary code.

This is not very efficient, but BIOSes are not usually masterpieces of code.

I don't have enough element to be sure what's going on, my point is that the ljmp 0xf000:0xfff0 and the mov al, byte es:[0xfff0] doesn't have to read from the same region they reside in.
With this in mind, all bets are off.
Only a proper reverse engineering will tell.

Regarding the wbinvd, I suggested in the comment it could be related to the warm boot facility and Peter Cordes suggested that it may specifically have to do with cache-as-RAM.
It makes sense, I guess will never be sure though.
It could as well be a case of cargo cult, where a programmer deemed the instruction necessary based rumors.

Margaret Bloom
  • 41,768
  • 5
  • 78
  • 124
  • Could you clarify why you think "the load mov al, byte es:[0xfff0] is not reading from the the BIOS first instruction, even though es is set to 0xf000"? In real-address mode, `es:[0xfff0]` is determined by `es*16 + 0xfff0`, i.e. `0xffff0`. Also, Intel Vol.3 Section 9.4.1 First Instruction Executed explicitly says "The first time the CS register is loaded with a new value after a hardware reset, the processor will follow the normal rule for address translation in real-address mode (that is, [CS base address = CS segment selector * 16])." So here the real-address mode is used. – user10865622 Jan 21 '19 at 00:42
  • @user10865622 : note that it says "first time the CS register is loaded with a new value". That only happens once a FAR JMP (or CALL) or equivalent type instruction is executed. When a modern 386 is booted the CS register has 0xF000 in it but that is just a display value as the hidden descriptor for CS is using a base of 0xFFFF0000 which is automatically added to the offset. This is similar to the [unreal mode](https://en.wikipedia.org/wiki/Unreal_mode) trick. – Michael Petch Jan 21 '19 at 00:47
  • @MichaelPetch Yeah, but it doesn't explain why "the load mov al, byte es:[0xfff0] is not reading from the the BIOS first instruction, even though es is set to 0xf000", the ES is assigned with a new value by `mov es, ax`. Also, even if it uses the base addr `0xffff0000`, the the calculated address will be `0xfffffff0`, which is the first instruction of BIOS. – user10865622 Jan 21 '19 at 00:49
  • Since ES is 0xF000 that move is 0xF000:0xFFF0 . However the BIOS entry point is actually at 0xFFFF0000+0xFFF0=0xFFFFFFF0 because the hidden descriptor cache for CS (and CS only) was preset with a base value of 0xFFFF0000 . On a 386 the BIOS entry point is not physical address 0xFFFF0 although you will find a copy of the BIOS there. – Michael Petch Jan 21 '19 at 01:03
  • @MichaelPetch Note there is a far jump `f000:f544 eaf0ff00f0 ljmp 0xf000:0xfff0`, which will modify the CS selector. Therefore, even if the code at `0xffff0` and `0xfffffff0` aren't the same (I highly doubt its possible), after one loop, the code is reading itself at the entry point `0xffff0`. Then the only explanation would be that the far jump at `0xffff0` is jumping to a different location, but that's really weird. – user10865622 Jan 21 '19 at 01:07
  • @user10865622 : That move to AL occurs before the FAR JMP. – Michael Petch Jan 21 '19 at 01:07
  • @MichaelPetch But the far jump is the following code flow for BSP if the `AL` moved is equal to `0xea`, so the BSP will execute that instruction. – user10865622 Jan 21 '19 at 01:11
  • 1
    @user10865622 : One way it may not be the same is if SMM modified it at boot. Wouldn't surprise me if somehow it has something to do with UEFI vs Legacy boot. SMM runs at a privilege level below 0 and can access the entire state of the processor and rewrite memory. In UEFI mode there would be no reason to necessarily have a real mode BIOS mapped into lower memory. – Michael Petch Jan 21 '19 at 01:11
  • 1
    @user10865622 If I was a betting man I'd guess that comparing 0x0000:0xfff0 with 0xea is the way that BIOS code determines whether it was booted as legacy or UEFI. If it thinks it is legacy mode it FAR Jumps to the realmode BIOS at 0xf000:0xfff0 to continue the legacy boot process on the BSP. Usually when using legacy mode the first instruction is a FAR JMP (not wbinvd and a relative jmp) with an instruction encoding of 0xEA . – Michael Petch Jan 21 '19 at 01:28
  • 1
    @user10865622 Based on your output, and my guess, it would mean your system is booting as UEFI (that guess may be entirely wrong). If it happens to be UEFI, what happens if you try booting Linux using legacy mode? Does that first instruction at physical address 0xFFFF0 then appear as a FAR JMP instead of wbinvd? – Michael Petch Jan 21 '19 at 01:31
  • In my comment above I mistyped `0x0000:0xfff0`.It should have been `0xF000:0xfff0` – Michael Petch Jan 21 '19 at 02:18
  • @user10865622 Sorry, I wnt to sleep and I lost track of the comments. Would you mind repeating your question? – Margaret Bloom Jan 21 '19 at 08:05
  • The question has been resolved. I misunderstood your first paragraph, I thought you were talking about GDT and selector in protected mode. It would be better if you explicitly say that `es:[0xfff0]` is reading `0xffff0` and the first BIOS code is at `0xfffffff0`, and most importantly, they can be different or even irrelevant. Now the code makes sense to me, the high memory code will delegate control to the legacy 1M memory if it finds a far jump at that position, though I have no idea how the ROMs are mapped to the address space. – user10865622 Jan 21 '19 at 09:37
  • @MichaelPetch I'm on UEFI and the hardware also supports legacy mode. What you guessed might be true, though I don't have enough disk to test a legacy boot now. – user10865622 Jan 21 '19 at 09:46
  • 1
    @user10865622 The iMC/PCH/North bridge has a register to control what happen to requests to the region `0xf0000-0xfffff`. They can go to the memory or they can be forwarded to the "IO subsystem". This can be done per request-type (read or write). The "IO subsystem" can be programmed to steers the requests in the `0xf0000-0xfffff` to the SPI/LLC flashrom. The memory near the top of the 4GiB resides in the PCI hole and it is directed to the "IO subsytem" that will, in turn, pass it to the flashrom. Basically it is an alias. This is used to shadow the bios by making the ram RO. – Margaret Bloom Jan 21 '19 at 09:49
  • @user10865622 : One way (without resorting to installing on HDD) to test might be to boot a live-cd of a popular Linux distro (with your system in legacy mode) use DD to acquire the bios.dump file. Mount a hard drive partition temporarily, copy bios.dump to it, unmount the har drive partition, reboot in UEFI mode from the HDD and then use radare2 to view the bios.dump file copied to the HDD partition. – Michael Petch Jan 21 '19 at 10:10
  • 4
    @MichaelPetch I finally found an old disk with a legacy linux installed, so I performed the test. Your guess is correct! When the machine is booted with legacy mode, the `0xffff0` code is completely different with the `0xfffffff0` code. The high memory code is the same as the one posted in question. The lower 1M BIOS starts with a far jump, to a different location, and the following code is also totally different. So this check is to differentiate legacy BIOS and UEFI mode. – user10865622 Jan 21 '19 at 13:44
  • @user10865622 Nice finding! Effectively, the fact that the higher range code is entering PM immediately was an indicator of UEFI. – Margaret Bloom Jan 21 '19 at 14:51
  • 2
    If you're still interested in this case, check out my answer. I think I've found a source that confirms your suspicion on this being a case of cargo cult. – wmjdgla Aug 10 '20 at 10:37
  • @wmjdgla Nice finding! – Margaret Bloom Aug 10 '20 at 11:57
4

This is actually the answer to the title question:

Hadi Brais: According to slide 14 of BIOS and System Management Mode Internals, the wbinv instruction was there in UDK2010 but then got later removed in UDK2012. Perhaps it's security-related. I don't know exact what.

I can confirm that this instruction is not present at 0xfffffff0 on my BIOS version from 2016.

There is a more burning question here and that's what does the comparison with comparison with 0xea mean.

The reset vector at 0xfffffff0 contains 90 90 E9 43 FC, which is a relative jump to 0xfffffff5-3bd which is 0xfffffc38, the entry point of my SEC Core PE32 image from 0xffffca18 - 0xffffffbb:

0x00:  DB E3                      fninit 
0x02:  0F 6E C0                   movd   mm0, eax   //move BIST value to mm0
0x05:  0F 31                      rdtsc  
0x07:  0F 6E EA                   movd   mm5, edx
0x0a:  0F 6E F0                   movd   mm6, eax  //save tsc
0x0d:  66 33 C0                   xor    eax, eax //clear eax

0x10:  8E C0                      mov    es, ax
0x12:  8C C8                      mov    ax, cs
0x14:  8E D8                      mov    ds, ax
0x16:  B8 00 F0                   mov    ax, 0xf000
0x19:  8E C0                      mov    es, ax
0x1b:  67 26 A0 F0 FF 00 00       mov    al, byte ptr es:[0xfff0]
0x22:  3C EA                      cmp    al, 0xea
0x24:  74 0E                      je     0x34   //if ea is at ffff0h then jump to the 0xf000e05b check 

0x26:  BA F9 0C                   mov    dx, 0xcf9
0x29:  EC                         in     al, dx    //read port 0xcf9
0x2a:  3C 04                      cmp    al, 4    
0x2c:  75 25                      jne    0x53      
0x2e:  BA F9 0C                   mov    dx, 0xcf9 //perform hard reset since if CPU only reset is issued not all MSRs are restored to their defaults
0x31:  B0 06                      mov    al, 6
0x33:  EE                         out    dx, al  

0x34:  67 66 26 A1 F1 FF 00 00    mov    eax, dword ptr es:[0xfff1]
0x3c:  66 3D 5B E0 00 F0          cmp    eax, 0xf000e05b
0x42:  75 0F                      jne    0x53      //if the ptr16:16 of the EA instruction isn't 0xf000e05b, move to notwarmstart

0x44:  B9 1B 00                   mov    cx, 0x1b //if it is equal, read bsp bit from apic_base msr
0x47:  0F 32                      rdmsr  
0x49:  F6 C4 01                   test   ah, 1
0x4c:  74 41                      je     0x8f   //if the and operation with 00000001b produces a zero result i.e. it's an AP then jump to cli, hlt

0x4e:  EA F0 FF 00 F0             ljmp   0xf000:0xfff0 //if it's the BSP and legacy bios is present, far jump to 0xffff0, exiting unreal mode

notwarmstart:
0x53:  B0 01                      mov    al, 1
0x55:  E6 80                      out    0x80, al  //send 1 as a debug POST code
0x57:  66 BE 68 FF FF FF          mov    esi, 0xffffff68
0x5d:  66 2E 0F 01 14             lgdt   cs:[si] //loads 32&16 GDT pointer (not 16&6, due to 66 prefix) at 16bit address fff68 in si into GDTR (base:ffffff28 limit:003f).

//enter 16 bit protected mode//
0x62:  0F 20 C0                   mov    eax, cr0
0x65:  66 83 C8 03                or     eax, 3   //Set PE bit (bit #0) & MP bit (bit #1)
0x69:  0F 22 C0                   mov    cr0, eax  //Activate protected mode
0x6c:  0F 20 E0                   mov    eax, cr4 
0x6f:  66 0D 00 06 00 00          or     eax, 0x600 //Set OSFXSR bit (bit #9) & OSXMMEXCPT bit (bit #10)
0x75:  0F 22 E0                   mov    cr4, eax

//set up selectors for 32 bit protected mode entry
0x78:  B8 18 00                   mov    ax, 0x18 //segment descriptor at 0x18 in GDT is (raw): 00cf93000000ffff
0x7b:  8E D8                      mov    ds, ax
0x7d:  8E C0                      mov    es, ax
0x7f:  8E E0                      mov    fs, ax
0x81:  8E E8                      mov    gs, ax
0x83:  8E D0                      mov    ss, ax
0x85:  66 BE 6E FF FF FF          mov    esi, 0xffffff6e
0x8b:  66 2E FF 2C                ljmp   cs:[si]   //transition to flat 32 bit protected mode and jump to address at 0x0:0xffffff6e aka. 0xffffff6e which is fffffcd8. CS contains 0 remember (it's the base that is 0xffff) so it will load the first entry.
                                                   //SEC continues at that address

0x8f:  FA                         cli    
0x90:  F4                         hlt    
.
.

We notice that my code differs from yours. There is an extra comparison to 0xf000e05b and a read/write to 0xcf9. Also, your system contains the e9 jump code we see at 0xfffffff0 at 0xffff0 as well, this is because you must be in UEFI boot and therefore there is no legacy BIOS shadowed to the legacy BIOS range at 0xf0000. Yours contains a wbinvd before the relative jump and mine contains 2 nops.

A clue here in the edk2 source code is that the code being jumped to is called 'NotWarmStart'. The code speaks for itself.

In mine, if EA is at 0xffff0 then it checks 0xffff1 for 0xf000e05b. If 0xf000e05b is there then it checks for the BSP flag, and if its the BSP, it jumps to 0xffff0. If 0xf000e05b isn't there, it jumps to the 16 bit + 32 bit protected mode setup (called 'NotWarmStart), which then jumps to 32 bit flat protected mode (edk2 calls this PEI, but I'd say PEI classically begins at the PEI core and that the code it jumps to is actually still SEC (given that this code uses FSP to set up CAR, optionally perform microcode updates if BootGuard isn't present and then passes control to the PEI core, and it's quite clearly an address in the code section of my SEC Core PE32 image, whereas PEI core image entry is at 0xffe91854)) implementation at 0x18:0xffffff6e. If EA is not present, it checks bit 3 of 0xcf9 for 'Check INIT# is asserted'. If it is asserted then it performs a warm reset, writing 0x6 which results in a PLTRST#, reason 'issue warm start, since if CPU only reset is issued not all MSRs are restored to their defaults'. If it isn't asserted then it jumps to 'NotWarmStart'.

On my system, both 0xffff0 and 0xfffffff0 are being redirected to SPI flash. You can disable the 0xffff0 range with BIOS_Legacy_F_EN, but when enabled, it always goes to where the range containing 0xfffffff0 is being redirected, you can't redirect one to SPI and another to LPC/eSPI. A cold legacy boot goes through SEC and PEI before it is extracted from the SPI flash and shadowed into RAM at 0xe0000 and/or 0xf0000 by redirecting writes to the memory controller in the Cbo SAD PAMs for those ranges but not reads, until the shadow is complete. On my system, the legacy bios is shadowed to 0xf0000.

My current guess is:

  • Containing 0xea and 0xf000e05b means that it is a legacy soft reset (CPU warm reset) or S3 resume and the legacy BIOS has already been shadowed into RAM, so let the legacy BIOS deal with the S3 resume or CPU warm reset.
  • Containing 0xea and not 0xf000e05b.... f000:e05b appears to be the POST entry point. Why it wouldn't contain that I don't 100% know, but of course, either the legacy BIOS or the UEFI bios implementation that shadowed in the legacy BIOS in the first place can change this, and when it's been changed to that it needs to go to SEC.
  • Not containing 0xea means either UEFI/legacy warm reset, UEFI soft reset, UEFI S3 resume or UEFI/legacy cold boot/reset, and if it is a soft reset, downgrade to warm reset, otherwise it needs to go to SEC.

Also there is no loop at hlt. Hlt enters a HALT C1 state, and responds to an INIT# IPI to put it in a wait-for-SIPI state. Execution will then begin at whatever address the BSP selects for the AP.

On my Windows 7 VirtualBox, which doesn't have a UEFI BIOS at all, both the 0xf0000 and 0xffff0000 map to the virtual LPC flash and therefore both show the BIOS.

kd> !db [uc] 00000000`ffffff80
#ffffff80 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
#ffffff90 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
#ffffffa0 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
#ffffffb0 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
#ffffffc0 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
#ffffffd0 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
#ffffffe0 00 00 00 00 00 00 00 00-00 00 00 00 00 00 58 4d ..............XM
#fffffff0 ea 5b e0 00 f0 30 36 2f-32 33 2f 39 39 00 fc 8f .[...06/23/99...
kd> !db [uc] fff80
#   fff80 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
#   fff90 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
#   fffa0 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
#   fffb0 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
#   fffc0 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
#   fffd0 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
#   fffe0 00 00 00 00 00 00 00 00-00 00 00 00 00 00 58 4d ..............XM
#   ffff0 ea 5b e0 00 f0 30 36 2f-32 33 2f 39 39 00 fc 8f .[...06/23/99...

On my legacy boot UEFI:

 ffffff80 FF 46 A0 70 CC CC CC CC E9 1B F6 FF FF E9 26 F6  Fáp¦¦¦¦Ú.÷  Ú&÷
 ffffff90 FF FF E9 71 F6 FF FF E9 7C F6 FF FF E9 3C FD FF   Úq÷  Ú|÷  Ú<² 
 ffffffa0 FF E9 3C FD FF FF E9 75 FF FF FF 00 00 00 00 00  Ú<²  Úu   .....
 ffffffb0 00 00 00 00 00 00 00 00 04 00 00 19 44 00 00 19 ............D...
 ffffffc0 00 01 D9 FF 00 00 00 00 00 00 00 00 00 00 00 00 ..+ ............
 ffffffd0 BF 50 41 EB 1D 00 00 00 00 00 00 00 00 00 00 00 +PAÙ............
 ffffffe0 54 18 E9 FF EB FE CF 00 00 00 00 00 00 00 00 00 T.Ú Ù¦¤.........
 fffffff0 90 90 E9 43 FC 00 00 00 FB 00 00 00 00 00 E9 FF ..ÚC³...¹.....Ú 

lkd> !db fff80
#   fff80 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
#   fff90 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
#   fffa0 00 00 00 00 00 00 00 00-00 00 e9 e3 3b 00 00 00 ............;...
#   fffb0 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
#   fffc0 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
#   fffd0 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
#   fffe0 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
#   ffff0 ea 5b e0 00 f0 30 37 2f-32 32 2f 31 36 00 fc 00 .[...07/22/16...
Lewis Kelsey
  • 4,129
  • 1
  • 32
  • 42
3

According to this Dr. Dobb's article written by Pete Dice of Intel back in 2011:

Because the processor cache is not enabled by default, it is not uncommon to flush cache in this step with a WBINV instruction. The WBINV is not needed on newer processors, but it doesn't hurt anything.

I'm not sure what WBINVD has to do with cache not being enabled by default. I thought he might have meant that WBINVD would enable the cache but the documentation doesn't say anything about the instruction having this effect. I think the second sentence confirms Margaret's suspicion that this is a case of cargo cult.

wmjdgla
  • 335
  • 2
  • 10
  • 3
    The processor's caches can never be truly completely disabled. Even with the caches disabled, any cache lines that still happen to be in the cache will continue to be used. Disabling the cache only prevents the CPU from adding new cache lines. The WBINVD instruction ensures that a disabled cache behaves like a disabled cache by ensuring there's nothing cached in the cache. It's possible there was some circumstance in older systems that the caches weren't reliably invalidated on reset, perhaps involving external caches, or the CPU's own internal self-test and initialization code. – Ross Ridge Aug 10 '20 at 15:59
1

I know x86 processor initialization always starts with a 16-bit 8086 environment, and the first instruction to be executed is at f000:fff0, i.e. 0xffff0.

No. For the last ~30 years segment registers have been split into a software visible part (e.g. the value pushed on the stack by a push cs) and several hidden internal fields which include a (32 bit) "base address". On power-on or reset that hidden "base address" part is set to 0xFFFF0000 and IP is 0xFFF0, which adds up to 0xFFFFFFF0; and that hidden "base address" part of CS won't change until something loads a new value into CS. However; typically firmware will switch to protected mode before that happens.

While in protected mode firmware will do a huge amount of stuff - configuring memory controllers (to get RAM working), figuring out PCI devices and device ROMs, building ACPI tables, etc. Part of that work is copying (maybe decompressing) a small "run-time" blob into the legacy area just below 1 MiB (which includes the instruction at 0xF000:0xFFF0 or 0x000FFFF0) and configuring the memory controller to pretend that the area of RAM is "write only" (so that it behaves like ROM even though it isn't ROM). In other words, millions of instructions might be executed before that "first" instruction at 0xF000:0xFFF0.

However; that whole legacy areas exists for backward compatibility with crusty old software from the 1980s (or earlier); and crusty old software from the 1980s may try to reset the computer by jumping to the instruction at 0xF000:0xFFF0 (under the "false for modern computers" assumption that it will begin the firmware's reconfiguration of everything - a full reset). With this in mind; the instruction at 0xF000:0xFFF0 is the start of code that pretends to reset whatever stuff crusty old software from the 1980s might know about (without bothering to reset/configure anything that crusty old software is too old to know about).

To my surprise, the first instruction is WBINVD, whose functionality is to invalidate the cache, which seems to be irrelevant when the processor is powered on or reset.

Right - if it was used for a power on or a real reset, the caches would be "disabled" and wbinvd would be irrelevant. It's not used for a real reset though, it's used for emulating a partial reset to keep crusty old software from the 1980s happy.

But it's not that simple - crusty old firmware expects caches to be disabled, so crusty old software would try to disable caches before jumping to the "pretend/emulated reset".

For modern CPUs you can't actually disable the caches. Instead, if you set the "cache disabled" bit in CR0 what you actually do is put it into a "no fill mode" where cache misses don't cause data to be inserted into the cache, but cache hits still work the same (data comes from the "not disabled" cache and doesn't get fetched from RAM). In this state (after setting the "cache disabled" bit in CR0) you need to do a wbinvd to flush the old contents of the cache (so that there are no cache hits) before you can achieve the behavior of "cache disabled". The reason the bit in CR0 is called "cache disable" is because for old CPUs it actually did disable the caches (if any).

Now you start to see the problem - for some CPUs (that crusty old software from the 1980s or earlier are likely to be expecting - primarily 80286 and 80386), the software can disable the cache (if any) just by setting a flag in CR0 then jump to 0xF000:0xFFF0 to start a "reset" (that, at least in theory, might expect caches to be disabled); and when this same crusty old software is run on a newer CPU it sets the flag in CR0 (which doesn't disable caches properly) and then something needs to do a wbinvd before the caches are actually disabled properly.

The end result should be obvious - the first instruction in the "legacy reset emulation that doesn't actually reset everything" is a wbinvd to achieve "cache disabled" behavior (that crusty old software was too old to do properly itself).

Mostly, I'd say there's lots of good reasons all this hacky mess got replaced by UEFI.. ;-)

Brendan
  • 35,656
  • 2
  • 39
  • 66
  • I mean now I think about it more, there shouldn't be a `wbinvd` there really . On mine there are 2 `nop`s. The SEC is supposed to be in L3 cache at this point, WBINVD will not write them back (and if anything is written back, you'd get an MCA) because theyre not modified, but it will invalidate the lines. Perhaps on his implementation SEC is not loaded into the L3 by the startup ACM and the intent is to disable the cache because `wbinvd` when `CR0.CD=1` disables the cache because all the lines are invalidated and thererefore nothing can hit in the cache, though I don't know why it needs to – Lewis Kelsey Apr 24 '21 at 14:32
  • Interesting that they chose `wbinvd` and not `invd` almost like there is some intention to cause an MCA if there is a modified line – Lewis Kelsey Apr 24 '21 at 14:48