0

I'm trying to write a program that prints the command-line arguments given to the executable. I've narrowed the problem down to this: When I call printf more than once, it prints the correct answer the first time and then the wrong answer on subsequent calls.

It is my understanding that 64-bit windows NASM passes parameters in the rcx, rdx, r8, and r9 registers. If this is the case, I don't know why I have to mess with the stack at the beginning and end of the function... I only know that a segfault occurs if I remove that code.

I've tried choosing different registers for storing the argc and argv params, including pushing them to the stack, but it always fails. Here's the simplest code I could write with the error:

global main
    extern  printf

section .data
    fmt_num: db '%ld Params...', 10, 0

section .text
main:
    ; Reserve local space on stack.
    push rbp
    mov rbp, rsp
    sub rsp, 8*4

    mov r11, rcx ; argc

    mov rcx, fmt_num
    mov rdx, r11
    call printf      ; Prints the correct number of parameters

    mov rcx, fmt_num
    mov rdx, r11     ; If I remove this line, it prints a seemingly random number.
    call printf      ; Consistently prints 582.

    ; Return local stack space.
    add rsp, 8*4
    mov rax, 0
    pop rbp

    ret

Changing the format string from %ld to %d has no effect, and changing the one line in the second call from:

mov rdx, r11

to something like:

mov rdx, 23

works just fine, so it's like the r11 register's value gets modified, but the issue still occurs even if I save and restore the register.

Digit
  • 68
  • 5
  • 5
    On Windows, and Linux, r11 is considered scratch, so printf is allowed to wipe it out, hence there's nothing surprising about this. https://www.dyncall.org/docs/manual/manualse11.html – Erik Eidt Mar 17 '23 at 17:30
  • Thank you, that explains why it consistently prints 582. However, the issue also occurs if I push r11 to the stack before the first call and then, after, I clear r11 and pop it from the stack. If I do that, the second call prints different numbers each time. – Digit Mar 17 '23 at 18:11
  • When the stack is unbalanced, pushing r11 and later popping won't necessarily help, because the pop doesn't pop what you're expecting, so, same problem. – Erik Eidt Mar 17 '23 at 18:12
  • 1
    The fix is to make sure the stack comes back to what you're expecting, at the time you're expecting it. If you push more than pop, the stack will become unbalanced, meaning the thing you think is at the top , isn't. This can easily happen with function calling. – Erik Eidt Mar 17 '23 at 18:14
  • Hopefully, you appreciate that doing `pop eax` puts the top stack item into `eax` rather than necessarily a value pushed by `push eax` — there's only one stack shared by all pushes & pops. – Erik Eidt Mar 17 '23 at 19:23
  • Using rcx and rdx for arg 1 and arg 2 is Windows convention. So is the creation of the shadow space (sub rsp, 8*4). The C function printf uses the shadow space. If you don't create the shadow space, you will overwrite the return address. You will also need to use the Windows preserved registers, rbx, rsi, rdi, r12, r13, r14, or r15 to avoid stacking the value. – hs takeuchi Mar 17 '23 at 20:10
  • @ErikEidt: the reason `push r11` / `call` / `pop r11` doesn't work is that function calls are also allowed to clobber their shadow space, 32 bytes above their return address. That's what the `sub rsp, 4*8` is reserving. You have to save it somewhere the callee isn't allowed to modify, like your own shadow space, or other stack space you allocated. [What is the 'shadow space' in x64 assembly?](https://stackoverflow.com/q/30190132) / [Shadow space example](https://stackoverflow.com/q/33273797) – Peter Cordes Mar 17 '23 at 20:45
  • Thanks, using r12 fixed the issue. Saving it to [rbp] also works. I'm still not sure I understand the shadow space but I'll definitely read up on it. – Digit Mar 17 '23 at 21:24
  • Saving it to `[rbp]` overwrites the saved RBP value that `main` restores before returning. RBP is call-preserved, so you're making `main` violate the calling convention. (The CRT caller apparently doesn't mind, so it happens to work anyway.) – Peter Cordes Mar 18 '23 at 01:10

0 Answers0