1

I wish to analyze assembly code that calls functions, and for each 'call' find out how many arguments are passed to the function. I assume that the target functions are not accessible to me, but only the calling code. I limit myself to code that was compiled with GCC only, and to System V ABI calling convention. I tried scanning back from each 'call' instruction, but I failed to find a good enough convention (e.g., where to stop scanning? what happen on two subsequent calls with the same arguments?). Assistance is highly appreciated.

Jaaz
  • 53
  • 6
  • 1
    GCC has two different strategies for calling a function like this. One is that it pushes the arguments on the stack and then cleans them up sometime after the function call, and the second is that reserves space for outgoing arguments of all the function calls at the start of the function and cleans it up once at the end. Either way the function arguments on the stack are volatile across the call, but only those arguments that are actually passed to that faction. That means an argument to a function can be put on the stack long before the call and across other calls. – Ross Ridge Dec 18 '16 at 21:05
  • 5
    You can't reliably tell in optimized code. And even doing a good job most of the time probably requires human-level AI. e.g. did a function leave a value in RSI because it's a second argument, or was it just using RSI as a scratch register while computing a value for RDI (the first argument)? As Ross says, gcc-generated code for stack-args calling-conventions have more obvious patterns, but still nothing easy to detect. – Peter Cordes Dec 18 '16 at 21:05
  • @PeterCordes Hmm... I was assuming a stack based calling convention, but yah, a register based one would make it completely impossible. – Ross Ridge Dec 18 '16 at 21:06
  • 1
    *what happen on two subsequent calls with the same arguments?* Compilers always re-write args before making another call, because they assume that functions clobber their args (even on the stack). The ABI says that functions "own" their args. Compiler-generated code that I've seen never does actually modify the stack memory holding its args, not even when that would enable a tail-call :( – Peter Cordes Dec 18 '16 at 21:07
  • This raises a conflict of supporting or not supporting GCC compiling optimizations. If not supporting optimizations, then the resulted code shall probably be more structured, yet if supporting optimizations I shall probably assume no situations where another reg is used as a scratchpad for the required reg, since it oftentimes requires extra instructions. – Jaaz Dec 19 '16 at 16:40
  • Yet if arguments are passed by the stack, then it shall probably be the easier case (and I conclude that all 6 registers are used as well). The real obstacle seems to be the case of registers only. – Jaaz Dec 19 '16 at 16:42

1 Answers1

4

Reposting my comments as an answer.

You can't reliably tell in optimized code. And even doing a good job most of the time probably requires human-level AI. e.g. did a function leave a value in RSI because it's a second argument, or was it just using RSI as a scratch register while computing a value for RDI (the first argument)? As Ross says, gcc-generated code for stack-args calling-conventions have more obvious patterns, but still nothing easy to detect.

It's also potentially hard to tell the difference between stores that spill locals to the stack vs. stores that store args to the stack (since gcc can and does use mov stores for stack-args sometimes: see -maccumulate-outgoing-args). One way to tell the difference is that locals will be reloaded later, but args are always assumed to be clobbered.

what happen on two subsequent calls with the same arguments?

Compilers always re-write args before making another call, because they assume that functions clobber their args (even on the stack). The ABI says that functions "own" their args. Compilers do make code that does this (see comments), but compiler-generated code isn't always willing to re-purpose the stack memory holding its args for storing completely different args in order to enable tail-call optimization. :( This is hand-wavey because I don't remember exactly what I've seen as far as missed tail-call optimization opportunities.

Yet if arguments are passed by the stack, then it shall probably be the easier case (and I conclude that all 6 registers are used as well).

Even that isn't reliable. The System V x86-64 ABI is not simple.

int foo(int, big_struct, int) would pass the two integer args in regs, but pass the big struct by value on the stack. FP args are also a major complication. You can't conclude that seeing stuff on the stack means that all 6 integer arg-passing slots are used.

The Windows x64 ABI is significantly different: For example, if the 2nd arg (after adding a hidden return-value pointer if needed) is integer/pointer, it always goes in RDX, regardless of whether the first arg went in RCX, XMM0, or on the stack. It also requires the caller to leave "shadow space".

So you might be able to come up with some heuristics to will work ok for un-optimized code. Even that will be hard to get right.

For optimized code generated by different compilers, I think it would be more work to implement anything even close to useful than you'd ever save by having it.

Community
  • 1
  • 1
Peter Cordes
  • 328,167
  • 45
  • 605
  • 847
  • 1
    It's actually not that hard to get the GCC to generate code that modifies the stack memory used to pass arguments. Just `void foo(int arg) { arg = 0;}` without optimization will do it. With optimization it requires that the compiler has to spill the register allocated for it back on the stack. Eg: https://godbolt.org/g/ogRl6n – Ross Ridge Dec 28 '16 at 06:28
  • @RossRidge: thanks. I thought gcc tended to allocate a local to hold values even if the incoming arg was "dead". Maybe I had never looked at actually modifying the C arg directly, since that's not how I usually write code. Hmm, even using a new C variable results in reusing the incoming arg stack slot. https://godbolt.org/g/8rRoxx. Cool, I guess I was wrong about that :) gcc 4.7 and older use a different strategy, and seems to be optimizing for CPUs where `push` is slow. I don't see any reuse of arg slots there. https://godbolt.org/g/nmRGnV – Peter Cordes Dec 28 '16 at 06:38
  • It seems to be different between x86 and x64, where in x64 GCC 6.3 will not reuse the stack argument. https://godbolt.org/g/Xo15f8 – Jaaz Dec 29 '16 at 06:52
  • @Jaaz: Sure, in 64-bit mode it doesn't need to, because there are enough call-preserved registers. gcc tends to favour saving/restoring a register for the function and using it to preserve things across function calls, instead of spilling/reloading its own values. This is a win for latency if the called function happens not to touch the register at all. – Peter Cordes Dec 29 '16 at 07:37