1

I'm new to assembly, and I am trying to write a simple program to produce a popup with some text in 64 bit assembly, using MASM64. I found a 32-bit version at https://www.bigmessowires.com/2015/10/06/assembly-language-windows-programming/, and have been trying to adapt it to 64 bit assembly.

The 32-bit original compiles fine with ml.exe, but of course produces errors with ml64. So far, I've removed the .686 .model flat stdcall at the start, changed End Main at the end to END, and changed push eax to push rax.

This is my code so far:

EXTERN MessageBoxA@16 : proc
EXTERN ExitProcess@4 : proc

.const
msgText db 'YES IT FINALLY WORKS!!!', 0
msgCaption db 'Hello World', 0

.code
Main:
push 0
push offset msgCaption
push offset msgText
push 0
call MessageBoxA@16
push rax
call ExitProcess@4

END

However, lines 11 and 12 (push offset msgCaption push offset msgText) keep producing this errror error A2070:invalid instruction operands.

I've looked everywhere and can't find why this doesn't work. What do I need to do differently in x64?

(I'm on Windows 10, Visual Studio 2017.)

UPDATE:

I changed push offset var to mov rax, offset var | puah rax and it compiled fine, and then I changed to fast calling convention as mentioned in the comments.

EXTERN MessageBoxA@16 : proc
EXTERN ExitProcess@4 : proc

.const
msgText db 'YES IT FINALLY WORKS!!!', 0
msgCaption db 'Hello World', 0

.code
WinMainCRTStartup: ;fast calling convention...
mov rcx, 0
mov rdx, offset msgText
mov r8, offset msgCaption
mov r9, 0
sub rsp, 32
call MessageBoxA@16
mov rcx, rax
sub rsp, 32
call ExitProcess@4

END

But I try to link it with
link /subsystem:windows /out:test64.exe kernel32.lib user32.lib test64.obj
and get this linker error...

test64.obj : error LNK2001: unresolved external symbol MessageBoxA@16
test64.obj : error LNK2001: unresolved external symbol ExitProcess@4
LINK : error LNK2001: unresolved external symbol WinMainCRTStartup
test64.exe : fatal error LNK1120: 3 unresolved externals

I've included kernel32.lib and user32.lib, and apparently the 64bit versions have the same name. I'm using the "x64 Native Tools Cmd Prompt for VS 2017" if that helps...

UPDATE 2:

Fixed by looking at code in duplicate link. (Why are x86 and x64 so different? Sigh)

Cody Gray - on strike
  • 239,200
  • 50
  • 490
  • 574
DarthVlader
  • 344
  • 3
  • 14
  • 1
    The 64 bit calling convention is totally different. See the [documentation](https://learn.microsoft.com/en-us/cpp/build/x64-calling-convention). – Jester Jan 02 '19 at 22:14
  • The code is contained in the question I marked as duplicate. – Jester Jan 02 '19 at 22:29
  • 1
    @Jester - this is more than a duplicate of the question linked to. digitalHamster0 - 64 bit immediate operands are only allowed with mov instructions. Otherwise, most immediate operands are signed 32 bit integers that get extended to 64 bits. To push an offset, you could use | mov rax, offset ... | push rax | or | lea rax,...| push rax | . As already commented, the 64 bit calling convention is more like fast call. For Windows, the first 4 input parameters are in RCX, RDX, R8, R9 (xmm registers may also be used). Also stack space is preallocated (typically 64 bytes) before making a call. – rcgldr Jan 02 '19 at 23:43
  • @zx485: I assume it won't even assemble because the offset is a 64-bit immediate that doesn't fit in 32 bits. Maybe Windows `.obj` files don't even have a relocation type that could represent an absolute address as a 32-bit sign-extended immediate, the way x86-64 Linux can (for position-dependent executables where the default non-PIE code model puts static symbols in the low 32 bits of virtual address space). – Peter Cordes Jan 03 '19 at 01:12
  • @rcgldr: Shadow space is 32 bytes (above the return address), not 64. Any stack args (that don't fit in registers) must start at RSP+32 (before `call` but after reserving shadow space). – Peter Cordes Jan 03 '19 at 01:14
  • @PeterCordes - OK, my prior comment was misleading. 32 bytes of shadow space for RCX, RDX, R8, R9 needs to be allocated | sub rsp,32 | prior to a call. Any stack based parameters need to be pushed onto the stack before allocating the 32 byte shadow space. However, I've seen examples like | sub rsp, 40 | mov [rsp+32],5th parameter | call ... | , and in some cases generic "headers" with | sub rsp,64 | ... | even when there aren't 4 stack based parameters being used. (Maybe something for debugger if not using frame pointers via RBP?) – rcgldr Jan 03 '19 at 03:48
  • @rcgldr: Reserving extra stack space also gives you some to use for locals that callees *won't* step on, if your own shadow space isn't big enough for everything you want to spill. But normally you want to `sub rsp, 16*n + 8`, unless you have an odd number of `push`es first. And no, for debugging / stack-unwinding there's metadata that describes the offsets where call-preserved registers have been saved. The format isn't so rigid. So yeah, `sub rsp,40` / `mov`-store makes perfect sense and looks normal. You don't have to literally `push`, just put them on the stack somehow of course. – Peter Cordes Jan 03 '19 at 03:56
  • @Jester Thanks, I didn't realise that. I have updated my code above – DarthVlader Jan 03 '19 at 10:25
  • It won't be `ExitProcess@4`, the `@4` was encoding how many bytes of stack space the callee popped (in the caller-pops `stdcall` convention). Since Windows x64 uses a caller-pops convention, I think those numbers are just gone. Look at compiler-generated code to see what asm symbol names it uses, as well as how to get the calling convention right. – Peter Cordes Jan 03 '19 at 10:34

0 Answers0