0

My code below works perfectly when I call the procedure from main. But if I call the procedure from a couple procedures deep, the stack accumulates return addresses, throwing off the location of my stack parameters. How can I modify my procedure to correctly allocate the proper shadow space before the call, no matter where in my code it is called?

Another way of putting this is that I call the CreateFile procedure several different places in my code, and each call requires a different number of bytes to subtract from rsp to make it work, due to the varying states of the stack. This is just not acceptable. I want to have a single procedure that works correctly no matter where it is called from.

mov rcx, pFileName
mov rdx, GENERIC_READ or GENERIC_WRITE
mov r8, FILE_SHARE_WRITE or FILE_SHARE_READ
xor r9, r9

sub rsp, 110o

mov rax, OPEN_EXISTING
mov qword ptr [rsp + 40o], rax
mov rax, FILE_FLAG_SEQUENTIAL_SCAN
mov qword ptr [rsp + 50o], rax
xor rax, rax
mov qword ptr [rsp + 60o], rax

call CreateFileA
add rsp, 110o
Peter Cordes
  • 328,167
  • 45
  • 605
  • 847
  • 2
    It's your job to always maintain proper stack alignment. If you do that you don't need to subtract different amounts. – Jester Mar 13 '16 at 13:22
  • 1
    @Jester's right - you need to maintain the stack. Every `call` instruction will push a return address onto the stack. If you're nesting calls, you are going to need to deal with the increasing stack depth. Relying on passing parameters on the stack in this situation may not be the best approach. At best, you may want to create a base pointer and pass that value to your function so you can use consistent offsets from a "fixed" base pointer, like how people traditionally use `rbp` to keep a fixed reference while manipulating stack and heap. – querist Mar 13 '16 at 13:42
  • 1
    x64 Masm Assembly requires passing params on the stack (must create shadow space) when passing more than 4 params. CreateFile requires 7 params, 4 passed in registers and 3 passed on the stack. –  Mar 13 '16 at 14:27
  • Also be aware that shadow space is a MASM thing only. Normal X64 assembly doesn't use it which is an annoying difference. Also you can pass 5 parameters in registers and, on windows only, have shadow space for them. – Careful Now Mar 13 '16 at 23:20
  • I've been using MASM for over 30 years, long before the other Assembly developers were even born! So, to me, MASM is the norm. But I know what you mean. Wish Microsoft supported MASM x64, but they don't. –  Mar 23 '16 at 19:22
  • I have figured out how to unwind the procedures and can now successfully write x64 MASM applications. –  Apr 09 '16 at 08:06
  • 1
    @CarefulNow: Windows x64 only allows 4 register args. Also, it's not about MASM vs. other assemblers, it's about the calling convention. Using NASM or GAS to make a Windows executable would still require shadow space when calling standard functions. If you were making a Linux executable that called libc functions, you'd be using the x86-64 System V ABI for those function calls, which doesn't use shadow space. MASM itself can only target Windows, but JWASM is compatible with its syntax and can assemble for other platforms. – Peter Cordes Sep 04 '22 at 15:22

1 Answers1

1

The normal way to handle this is to keep the stack aligned. That requires code inside every non-leaf function, to move RSP by an odd multiple of 8 including any pushes you do. You also have to reserve an extra 32-bytes, so typically you'd sub rsp, 40 at the top of a function if you didn't push anything.

Don't use that bottom 32 bytes of stack space in any non-leaf function. That space becomes shadow space for any functions you call.

You don't need to modify RSP around each call within a function, only at the start/end of a function.

Look at compiler-generated code for examples, such as MSVC on https://godbolt.org/ ; see also How to remove "noise" from GCC/clang assembly output?

Or for example the GCC output in Is the caller or callee responsible for freeing shadow store in x64 assembly (windows)? - that makes two function calls back to back, call printf and call bar, with no need for any instructions between them, because bar takes no args.

Peter Cordes
  • 328,167
  • 45
  • 605
  • 847