4

I'm trying to figure out some of the SSE instruction set's operand types in virtual real mode / real mode / 16-bit protected mode.

For example let's take the following instruction:

66 0F 38 01 /r PHADDW xmm1, xmm2/m128

The destination operand is an XMM register and the source operand is an XMM register or a 128-bit memory.

Now I couldn't find anything regarding the modes I've been mentioning in the Intel reference and the SSE instruction's operand types changing in these modes, but according to capstone this instruction's source operand suddenly became a 64-bit memory operand, not to mention that it seems capstone just dismissed the mandatory 0x66 operand size prefix for this instruction and used the addressing P and Q instead of V and W (as mentioned in the Intel ref) for the destination and source operands accordingly, and thus disassembling it as:

66 0F 38 01 58 D6    phaddw mm3, qword ptr [bx + si - 0x2a]

with an MMX register instead of an XMM register as I would have thought.

To my understanding the correct disassembly should be:

phaddw xmm3, xmmword ptr [bx + si - 0x2a]

using 128-bit memory as the source operand, and using an XMM register as the destination operand.

So either capstone is wrong or I'm missing something regarding SSE instructions and 16-bit mode somewhere in the Intel ref.

Clarification would be appreciated.

Sep Roland
  • 33,889
  • 7
  • 43
  • 76
Jorayen
  • 1,737
  • 2
  • 21
  • 52
  • 3
    You convinced me :) Also both `objdump` and `ndisasm` produce the expected result. – Jester Dec 24 '22 at 23:30
  • 4
    "Legacy SSE" works in real mode (AVX/anything with a VEX prefix doesn't), so capstone has to be wrong – harold Dec 24 '22 at 23:41
  • @harold I see thanks for clarifying, I could find more legacy sse instructions that capstone doing funny business with which I couldn't explain, so I would now feel more comfortable assuming capstone is in the wrong here. – Jorayen Dec 24 '22 at 23:50
  • IIRC, I think even AVX works in 16-bit *protected* mode. It's only real mode where VEX instructions fault as the [invalid encoding of LES / LDS they overlap with](https://stackoverflow.com/questions/68898858/how-does-an-instruction-decoder-tell), because apparently some VM86 software (like NTVDM) also used that intentionally to trap to protected mode or to check for a VM. [See Michael Petch's comment](https://stackoverflow.com/posts/comments/63139570) @Harold is correct that legacy SSE encodings work in real mode while VEX encodings don't (including BMI1 `andn` and similar). – Peter Cordes Dec 25 '22 at 00:52
  • 1
    @Jester I reported similar [NDISASM bug #1](https://forum.nasm.us/index.php?topic=2463.msg11129#msg11129) concerning incorrect disassembly of ModR/M in 16bit mode. – vitsoft Dec 25 '22 at 12:26

0 Answers0