3

I am writing a VMM, and I'm trying to support virtual accesses to the x2APIC's registers by a guest OS running in VMX non-root mode.

I want to start off by doing something simple, such as reading the local APIC ID from within the guest OS. I've tried adding support for this in my VMM, but the value I read seems to be incorrect.

Unfortunately, I can't seem to find a lot of information online about the virtual APIC page. I've read through chapter 29 of the Intel manual (APIC Virtualization and Virtual Interrupts), and here's what I'm doing:

  1. In the secondary processor-based VM-execution controls, I set the following bits to 1: (I'm setting bit 9 below since I ultimately want to support posted IPIs)

    1. SECONDARY_EXEC_VIRTUALIZE_X2APIC_MODE (bit 4)
    2. SECONDARY_EXEC_APIC_REGISTER_VIRT (bit 8)
    3. SECONDARY_EXEC_VIRTUAL_INTR_DELIVERY (bit 9)

  2. In the MSR bitmap, I disable intercepts for 0x802, which is the local APIC ID register.

  3. In my guest OS, I use rdmsr to read 0x802.

I perform step 3 on two threads that are pinned to different cores. They both read the value 2621447225 from the register. This seems incorrect since the threads are pinned to different cores and should therefore read different local APIC IDs (and the number 2621447225 is really big). What am I doing incorrectly?

Here's some additional information for your reference:

In section 29.5 (Virtualizing MSR-based APIC Accesses) of the Intel manual, it says:

If “APIC-register virtualization” is 1 and ECX contains a value in the range 800H–8FFH, the instruction reads the 8 bytes from offset X on the virtual-APIC page into EDX:EAX, where X = (ECX & FFH) « 4. This occurs even if the local APIC is not in x2APIC mode (no general-protection fault occurs because the local APIC is not in x2APIC mode).

The X offset makes sense to me: the MSR address 0x802 will be 0x2 when AND'd with 0xFF, and 0x2 will become 0x20 when left-shifted 4 bits. 0x20 is the offset within the physical APIC's page if you were accessing an xAPIC through its memory-mapped registers. 8 bytes (i.e. 64 bits) are then read, so the lower 32 bits are the local APIC ID for x2APIC.

Jack Humphries
  • 13,056
  • 14
  • 84
  • 125
  • 2
    Do you initialize the virtual APIC page with the values of the virtual APIC registers? – prl Dec 23 '17 at 08:06
  • @prl Ugh nope, I didn't know that I needed to do that. Sorry - the division of labor between the VMM and the CPU is unclear to me, so I didn't know that I needed to write this in myself. Let me get that working and I'll report back. Thanks a lot by the way :) – Jack Humphries Dec 23 '17 at 08:11
  • 1
    And anytime the state of the virtual APIC (in your VMM) changes, it needs to update the appropriate registers in the virtual APIC page. – prl Dec 23 '17 at 08:15
  • 1
    @prl Perfect! This worked great. Everything works now. Feel free to write an answer so I can give you points, or I'll write one myself if your prefer. – Jack Humphries Dec 23 '17 at 23:29
  • @prl One more (hopefully quick) question: I'm now trying to write 64 bits to the ICR using `wrmsr` with address `0x830`. I've set up the virtual-APIC page in the VMCS, but not the APIC-access page since I'm using x2APIC. Since I'm sending an IPI to another core, I expect that I would need to emulate this write myself in the VMM. Unfortunately, when I execute `wrmsr`, I get a GP fault even though I am expecting APIC write exit. Any ideas regarding this? I configured the MSR bitmap to not cause an intercept on `0x830`. – Jack Humphries Dec 23 '17 at 23:32
  • I think the GP is because you are setting a reserved bit in the ICR. – prl Dec 24 '17 at 01:48
  • I don't think you can get an APIC-write exit for an access to MSR 830 in x2apic mode. Special processing only applies to 808, 80b, and 83f. For 830, if the corresponding bit in the MSR bitmap is 0, it accesses the actual APIC register, allowing guest software to directly issue an IPI. Probably not what you want, so you need to set the bit in the MSR bitmap to 1 to get a VM exit. – prl Dec 24 '17 at 01:48
  • Note, I haven't actually implemented this myself. My comments are based on the information in section 29.5. – prl Dec 24 '17 at 01:50
  • @prl I looked through the source for KVM and you appear to be right. They intercept all MSR accesses to the x2APIC's registers and then handle them in KVM. – Jack Humphries Dec 30 '17 at 02:07

1 Answers1

1

I was able to figure this out with the help of @prl. I had to allocate a virtual-APIC page myself for each core and then initialize each page individually with its associated core's local APIC ID.

I then added the physical address of the page to the VMCS (there's a constant defined in the Linux kernel called VIRTUAL_APIC_PAGE_ADDR that contains the offset within the VMCS). I didn't realize that I had to initialize the page since it wasn't done automatically.

Edit: I implemented a working virtualization system on Linux with support for x2APIC virtualization and posted interrupt processing, and wrote about those two topics in this document. The document should be fairly straightforward, and it contains a link to my implementation.

Jack Humphries
  • 13,056
  • 14
  • 84
  • 125