"undefined symbol rax, rbx, rcx, rdx" in MASM Visual Studio

Question

Following is the code:

.386

.model flat, stdcall

.stack 4096

    ExitProcess PROTO, dwExitCode:DWORD

    .DATA

    .code

Main proc

mov rax,0

mov rbx,0

mov rcx,0

mov rdx,0

       mov al, 0F0H           ;using 8-bit register

    mov bx, 1012H            ;using 16-bit register

    mov ecx, 15284H          ;using 32-bit register
   
    mov rdx, 3578815H        ;using 64-bit register


INVOKE ExitProcess, 0

main ENDP

END main

Whenever I run this, it says:

undefined symbol rax, rbx, rcx, rdx

But it works when I use:

.386
.model flat, stdcall
.stack 4096
    ExitProcess PROTO, dwExitCode:DWORD
    .DATA
    .code
Main proc

mov eax,0

mov ebx,0

mov ecx,0

mov edx,0

       mov al, 0F0H           ;using 8-bit register

    mov bx, 1012H            ;using 16-bit register

    mov ecx, 15284H          ;using 32-bit register  
 
    mov edx, 3578815H        ;using 64-bit register  

INVOKE ExitProcess, 0

main ENDP

END main

I'm using x86 configuration. x64 doesn't work for me. Could someone tell me why RAX, RBX, RCX, and RDX aren't working?

So youre saying I can't see how 64 bit registers are working and it's only limited to 32- bit registers or lower? Is there surely no other way to see how 64 bit registers are working? — The Channel, Oct 07 '22 at 10:56
@TheChannel Specifying `.386` directive you limited assembler by 80386 capabilities. To use 64 bit registers you have to specify target which has ones. I guess MASM has directives or command line options for x86_64. Sorry, i'm not familiar with modern MASM. — dimich, Oct 07 '22 at 11:14

score 2 · Answer 1 · answered Oct 07 '22 at 13:09

2

For Visual Studio, a 64 bit build would use ML64.EXE instead of MASM.EXE. Don't use .386 directive. There is no .model directive for ML64.EXE, since the 64 bit calling convention is fixed.

answered Oct 07 '22 at 13:09

rcgldr

27,407
3
36
61

Peter Cordes · Answer 2 · 2022-10-07T18:42:21.070

You can't use 64-bit registers in 32-bit code.

In 16-bit mode, 32-bit registers are usable in machine code with 66h and 67h prefixes. So 16 and 32-bit code can both use instructions like movzx ax, byte ptr [ecx] (assuming the code is running on a 386 or newer, of course, hence .model 386 telling the assembler that it should let you write code like that).

But for x86-64, there wasn't an unused byte left for use as a prefix in 32-bit mode, so the AMD64 architects had to do it differently. The new features of x86-64 are only available in 64-bit mode, not in 32-bit compat mode.

The actual mechanism they picked was to remove the 1-byte encodings for inc reg and dec reg, and reuse those 0x4? bytes as REX prefixes, including a Width bit to distinguish add eax, ecx from add rax, rcx, and 3 extra bits for register numbers, allowing access to r8-r15 for instructions like add r8d, [r9 + r10].

As https://wiki.osdev.org/X86-64_Instruction_Encoding#REX_prefix says, The REX prefix is only available in long mode.

Fun fact: If you try to execute 64-bit machine code with the CPU in 32-bit mode (e.g. from using NASM bits 64 in the source and assembling with nasm -fwin32), the REX prefixes will decode as inc or dec instructions.

You can play fun tricks like making machine code that runs 3 different ways depending on what mode it executes in. For example, x86-32 / x86-64 polyglot machine-code fragment that detects 64bit mode at run-time?. Also a 3-way codegolf answer shows disassembly of the same code 3 different ways.

"undefined symbol rax, rbx, rcx, rdx" in MASM Visual Studio

2 Answers2