x64 function detouring without inline assmebly

Question

Good evening, fellow coders and hackers.

I'm experimenting with binary patching, more precise: detouring functions that are not called via vftable.

What I am doing in detail

I'm injecting a DLL into a running process and determine the start address of a function (original function) by scanning for it signature.

Once I've found it, I rewrite the first 13 bytes with my own shell code to redirect every call to this function to a function of my DLL (hook function):

mov rax, <dll_function_address>
jmp rax
ret

The hook function calls the address of the original function again, after removing my shellcode in order to prevent an endless recursion. Once the original function returns, the hook function is meant to return the value it has returned to preserve the regular code flow.

The issue

You might have noticed that my shellcode does by no means preserve any registers (not even RAX, which might not only be manipulated by the hook function, but is already manipulated by my shellcode.

Therefore, the original function fails when called by the hook function. I wanted to add inline assembly to push the registers to the stack as very first action in the hook function as well as popping them back before passing control to the original function, but Visual Studio does not have x64 support for inline assembly.

I could supplement my shellcode with opcodes to push the registers to the stack. But I can't add opcodes to pop them back because I restore the original code before calling the original function

The solution I'm trying to circumvent

I actually know that the most clean approach for this problem is to relocate the whole function. This way, I wouldn't be forced to remove the hook before calling the original function. I could add opcodes to push the registers to my jmp shellcode and opcodes to pop them infront of the relocated function. I'm trying to circumvent this because I don't know how to dynamically determine the end of the original function, so I cant figure out how many bytes to move.

Questions

1. Is there some kind of calling convention or something else I'm not yet aware of, that would allow me to declare my hook function in a way that would force the compiler not to use registers, but only the stack?
2. Does somebody know how I can tell whether a 0xC3 (ret) byte is actually the function end?
3. Could I configure Visual Studio 2013 to use alternative compilers? With x64 inline assembly support?

Answer 1

One solution would be, instead of restoring the original function from your hook is to call the same instructions that you overwritten from your hook, example:

Original function:

<do_stuff>
  push rbp
  mov rsp, rbp
  sub 8, rsp
  add rbx, rcx
  ....

Hooked function:

<do_stuff>
  mov rax, <dll_function_address>
  jmp rax
  nop
  add rbx, rcx
  ...

Your dll function:

  ... // Do your hooked stuff
  push rbp // Repeat the code your replaced from the hooked function
  mov rsp, rbp
  sub 8, rsp
  jmp <do_stuff + sizeof(shellcode) + nopsled> // Call it and add the offset of the hook shellcode

If the shell code doesn't align with the instruction you can nop sled as shown in the example.

As for the rax problem, I think you can do something like this:

<do_stuff>
  push <dll_function_address>
  ret
  nop
  add rbx, rcx
  ...

It pushes your hook address on the stack and call ret, ret take the first address on the stack and jump there.

Or maybe you can use call instruction:

<do_stuff>
  call <dll_function_address>
  nop
  add rbx, rcx
  ...

Once the hook is called, replace the hooked function as you already do. Then the remaining issue is that the ret from your dll function will continue on the nop instruction, so you need to remove 5 (the size of the call instruction) from the adress pushed in the stack, to do that, I found this: https://stackoverflow.com/a/12631122/2838914.