0

I am working with Windows and x64 assembly. I am eager to know more and more about x64 calling convention. My program seems couldn't handle calling convention correctly, so I need some help and experience to handle its issue.

Based on MSDN:

The x64 Application Binary Interface (ABI) uses a four-register fast-call calling convention by default. Space is allocated on the call stack as a shadow store for callees to save those registers.

So before I make any call to Windows API, I try to store all parameters in the registers RCX, RDX, R8 and R9 but when I wanted to call WriteConsoleA, It get 5 parameter from user so I decided to add fifth parameter into the stack. Also, I consider when Call instruction trigger it put return address of the caller to the stack so the ret instruction work correctly.

However, based on this assumption, I written the following code. It is work and print HelloWorld to the screen but when WriteConsole triggered it can't back to the caller position and successfully call ExitProcess function. Because when program done, in console I see the return value of the program isn't zero.

This is the final code which I have written as a learning process:

GetStdHandle PROTO
ExitProcess PROTO
WriteConsoleA PROTO

.data
    message         DB "Hello World",0
    message_size    DQ SIZEOF message

.code
    ShowMessageConsole PROC
        POP R10
        POP R9
        POP R8
        POP RDX
        POP RCX
        MOV  QWORD PTR [RSP + 4 * SIZEOF QWORD], 0
        CALL WriteConsoleA
        RET
    ShowMessageConsole ENDP

    main PROC
        MOV RCX, -11           
        CALL GetStdHandle
        PUSH RAX
        LEA RDX, message
        PUSH RDX
        MOV R8, SIZEOF message - 1
        PUSH R8
        MOV R9, message_size  
        PUSH R9
        CALL ShowMessageConsole

        MOV RCX, 0      
        CALL ExitProcess
    main ENDP
END

This program show Hello World to the screen, but terminated with a random value so I can understand program execution flow corrupted. I debug the program but I couldn't figure out the problem of the program.

Thank you folks for helping me and other people who entered to the low-level field of computer system. Thank you stackoverflow for this nice platform.

Homayoon
  • 13
  • 3
  • 4
    Write it in C language. Compare genrated code with your one and you will get the answer without guessing. Probably faster than writing the question here – 0___________ Jan 05 '23 at 10:57
  • `POP R10` pops your return address. Earlier function calls pop space that's been part of the shadow space for WinAPI calls. The `ret` after `CALL WriteConsoleA` is I think using `main`'s return address if I counted pushes and pops correctly, and if WriteConsoleA happens not to overwrite that part of its shadow space. If so, your program effectively does `return WriteConsoleA(...)`. Read up about what Shadow Space is, then single-step with a debugger. – Peter Cordes Jan 05 '23 at 11:34
  • Thanks Peter, Where can I find out more information about shadow space? @PeterCordes – Homayoon Jan 05 '23 at 12:20
  • The same MS docs you already found, and/or google, e.g. for `site:stackoverflow.com x64 shadow space`. e.g. [Shadow space example](https://stackoverflow.com/q/33273797) / ["Hello world" in x64 assembly for Windows - Shadow space / Stack alignment](https://stackoverflow.com/q/73420722) / [What is the 'shadow space' in x64 assembly?](https://stackoverflow.com/q/30190132) – Peter Cordes Jan 05 '23 at 12:21

0 Answers0