0

I'm writing a tiny library in C and assembly. I'm nearly done the linux version. I build with -static -nostdlib and it was trivial to get windows to build the linux binary. On linux I use assembly to get argc+argv for main and to implement linux calls via syscall so I can do mmap, open/write/read/close, etc.

Now I'd like to write the assembly for windows. From my understanding I don't want to do syscalls or rely on hardcoded addresses. While researching I found a few examples using NASM and MASM to call window functions by specifying the function name in the assembly file. I'd like to write assembly compatible with clang and I'm pretty new to assembly

My question is how do I call OpenFile and VirtualAlloc from an assembly file that clang understands? And what build options might I need? I'll eventually want to build the windows binaries from linux but I'll start with building on windows until I get the assembly and the rest of my code working

Eric Stotch
  • 141
  • 4
  • 19
  • 1
    Semi-related: you had mentioned wanting to write pure asm stuff with minimal libraries on Linux. A Q&A about the Windows equivalent is [FASM write Hello World to console with NO includes or dependencies at all](https://stackoverflow.com/q/63388863). Apparently the Windows program-loader *always* maps a couple DLLs into a process's address-space even if they're not explicitly mentioned. (I forget if answers/comments there are where I learned that, or somewhere else.) – Peter Cordes Oct 12 '21 at 12:21

1 Answers1

1

Invokation of system functions on 64bit Windows requires to mark the function name as imported from its DLL, align RSP, push or load its arguments, create shadow space, call the function name and finally restore RSP to equilibrum (to the same value which it had before the invokation).

Function OpenFile is deprecated, MS recommends CreateFileA instead.
Look at Example: Fast version of invokation WinABI: The following example is in Intel-syntax, asm-able on Windows with euroasm.exe example.asm. It may need some modifications for other assemblers: SPL is called SPB in some other assemblers, which also may require default rel to prefer RIP-relative encoding of LEA and ADD qword [RSP],8 rather than ADDQ [RSP],8. If your tool prefers AT&T syntax instead of Intel syntax, change the order of ASM operands and prefix register names with %.

IMPORT CreateFileA, LIB="kernel32.dll" 

 PUSH RSP                    ; Store original stack pointer value (equilibrum).
 TEST SPL,1000b              ; Test RSP OWORD alignment at run-time.
 JNZ .WinABI1                ; Skip to the label when not aligned.
 PUSH RSP                    ; Store and update 2nd copy of original RSP (equilibrum).
 ADDQ [RSP],8                ; Those two instructions aren't executed if RSP was properly aligned.
.WinABI1:                    ; Local label. 
 PUSHQ 0                     ; Push 7th argument.
 PUSHQ FILE_ATTRIBUTE_NORMAL ; Push 6th argument.
 PUSHQ OPEN_EXISTING         ; Push 5th argument.
 MOV R9,0                    ; Load 4th argument.
 MOV R8,FILE_SHARE_READ      ; Load 3rd argument.
 MOV RDX,GENERIC_READ        ; Load 2nd argument.
 LEA RCX,[FileName]          ; Load 1st argument.
 SUB RSP,4*8                 ; Make room for shadow space in fast mode. RSP is OWORD-aligned.
 CALL CreateFileA            ; Call the imported function.
 LEA RSP,[RSP+7*8]           ; Discard transferred arguments, keep RFlags.
 POP RSP                     ; Restore RSP to equilibrum from 1st or 2nd copy.

Unlike on Linux, if you want to allow nonASCII characters in FileName, they are expected in wide (UTF-16) encoding, and CreateFileW should be invoked instead of CreateFileA.
Although the calling convention is very different in 64bit version of Windows, functions are still being imported from %SystemRoot%\System32\kernel32.dll, %SystemRoot%\System32\user32.dll etc.

vitsoft
  • 5,515
  • 1
  • 18
  • 31
  • Is this supposed to assemble + link with clang / GNU assembler? It doesn't look like it, no `.intel_syntax noprefix`, and missing `[RIP + FileName]` for the addressing mode. Is this EuroAssembler syntax instead? If so, say so; the whole point of this Q&A is that the examples Eric was finding don't work with pure clang. – Peter Cordes Oct 12 '21 at 19:31
  • Also, the other key part of the question is what exact commands can you run (on Linux) to build a Windows executable. So a valid answer will need build commands as well as source, even if you choose to answer with a different assembler than the one asked for. – Peter Cordes Oct 12 '21 at 19:37
  • 1
    @PeterCordes Eric does not know much assembly and I don't know clang, so I wanted to help with only a part of his question: how to call a Windows function from assembly. Yes, the example is in €ASM syntax, but generic enough to be used by other tools with minor adjustment (I hope). – vitsoft Oct 12 '21 at 20:06
  • 1
    Yeah, there's certainly some value in an answer like this. But it would be much more useful if you included the necessary build commands (which would make it clear that this *isn't* GAS / clang syntax; you didn't even mention that in your current answer! Asm beginners aren't going to be able to identify the necessary tools from looking at the code). – Peter Cordes Oct 12 '21 at 20:09