1

I am trying to get the parameters of a 64-Bit __fastcall function, but I am having a couple of issues/questions.

1) I checked the registers in the debugger and when I have 3 32-bit parameters and a void function, the second one goes into RDX, the third one into R8 and the first one I cannot see at all and assume is on the stack. I did not check every possible combination but this goes against what MSDN's documentation on 64-bit __fastcall says. ...Or am I missing something?

-- Regarding 1 I just realized I think it says that if I pass a 32-bit value into a 64-bit register it's not 0 extended so I probably just missed it due to gibberish data that was in the RCX register.

  1. Due to VS not support 64-bit inline assembly or any useful intrinsics (At least that I can find), I wrote a shellcode to get all of the parameters from RCX, RDX, R8, R9, XMM0-3.

The issue here is that in order to prepare the shellcode I have to allocate memory, copy memory then set the EIP to my shellcode or calling it, etc. which screws up the thread's context. Is there any way to cleanly do this?

Peter Cordes
  • 328,167
  • 45
  • 605
  • 847
Carol Victor
  • 331
  • 1
  • 7
  • "_`__fastcall` - This calling convention only applies to the x86 architecture_" - From [`__fastcall`](https://learn.microsoft.com/en-us/cpp/cpp/fastcall?view=msvc-160) – Ted Lyngmo Jan 20 '21 at 17:28
  • 1
    `Given the expanded register set, x64 uses the __fastcall calling convention and a RISC-based exception-handling model` https://learn.microsoft.com/en-us/cpp/build/x64-software-conventions?view=msvc-160 .. Windows's VS turns every other calling convention into fastcall in 64-bit mode – Carol Victor Jan 20 '21 at 17:29
  • 3
    There seems to be a conflict in what they are saying - and those pages were updated on the same day :) "_The `__fastcall` keyword is accepted and ignored by the compilers that target ARM and x64 architectures; on an x64 chip, by convention, the first four arguments are passed in registers when possible, and additional arguments are passed on the stack._". [x64 calling convention](https://learn.microsoft.com/en-us/cpp/build/x64-calling-convention?view=msvc-160) was updated more recently. – Ted Lyngmo Jan 20 '21 at 17:30
  • It's possible that I got the names confused due to MS' documentation, but regardless of the name of the calling convention, I followed their documentation regarding how parameters are passed in 64-bit (rcx, rdx, r8, r9, xmm0-3 for fp and everything else on stack). My second question is also unrelated to the first, but they both relate to the topic of variadic functions in 64-bit – Carol Victor Jan 20 '21 at 17:35
  • @TedLyngmo seems worthy of an answer? – SergeyA Jan 20 '21 at 17:38
  • @SergeyA I'm just reading from MSDN and can't really add anything that's not there. My assembly mojo is broken :-) There _is_ an interesting paragraph in [x64 calling convention](https://learn.microsoft.com/en-us/cpp/build/x64-calling-convention?view=msvc-160) especially mentioning vararg functions. (no idea about the downvotes - seems like a legit question to me) – Ted Lyngmo Jan 20 '21 at 17:42
  • 1
    (cont.) "_If parameters are passed via varargs (for example, ellipsis arguments), then the normal register parameter passing convention applies. That convention includes spilling the fifth and later arguments to the stack. It's the callee's responsibility to dump arguments that have their address taken. For floating-point values only, both the integer register and the floating-point register must contain the value, in case the callee expects the value in the integer registers._" – Ted Lyngmo Jan 20 '21 at 17:46
  • I'd like some input on the second issue. If I get that working I'll eventually figure out how the 64-bit calling convention passes the parameters. It is quite unfortunate there is still no inline assembly in 64-bit, it's like they gave up half-way trough by just releasing 32-bit inline assembly. – Carol Victor Jan 20 '21 at 17:56
  • 1
    @TedLyngmo: The Windows x64 default calling convention is called `__fastcall` by MS's documentation in a couple places, including [here](https://learn.microsoft.com/en-us/cpp/build/x64-software-conventions?view=msvc-160). (As opposed to `__vectorcall`, which is like fastcall but with __m128 and wider args allowed to be in an XMM register instead of passed by reference to keep everything fixed-width). And the 2020-updated page you linked describes it as "a four-register fast-call calling convention by default". So IMO fastcall is not wrong as a name for the Windows x64 convention. – Peter Cordes Jan 20 '21 at 18:54
  • 1
    @CarolVictor: MSVC's compiler internals that handle inline asm are apparently so brittle and clunky that they couldn't make it safely handle anything but 32-bit. e.g. it's not safe to use MSVC inline asm in functions with register args, but in x86-64 that can only happen if you have no args. If you want to write asm by hand, do it in a `.asm` file, or just look at compiler asm output. ([How to remove "noise" from GCC/clang assembly output?](https://stackoverflow.com/q/38552116) 's pointer to https://godbolt.org/ applies to MSVC as well) – Peter Cordes Jan 20 '21 at 18:57
  • 1
    Related: my answer on [How to set function arguments in assembly during runtime in a 64bit application on Windows?](https://stackoverflow.com/q/49283616) takes advantage of the documented behaviour of the simplicity of Windows x64 by putting the first 4 args into both integer and XMM regs, so it works regardless of type. As I said there, this is already required for variadic functions so the callee can just dump its integer regs into the shadow space to have once contiguous array of args (continuing on into the stack args if any). – Peter Cordes Jan 20 '21 at 19:04
  • 1
    Raymond Chen's answer on [x64 calling convention (stack) and varargs](https://stackoverflow.com/q/12083810) shows a diagram of stack layout,. – Peter Cordes Jan 20 '21 at 19:05

0 Answers0