2

I believe I understand the difference between STDCALL and CDECL but I'm wondering if I can find some clarification within this code.

I understand that in STDCALL the CALLEE is responsible for cleaning up the stack, and I understand that in CDECL the CALLER is responsible for cleaning up the stack.

I also understand that "cleaning up the stack" basically means re-setting the stack pointer, but I guess my confusion comes in at this line of code where the value of esp is being moved into ebp, the base pointer. If that function is happening, is that the same thing as "cleaning up the stack" ? Or does it have to be something moving into ESP specifically?

Here is the code I'm looking at

main PROC
    push 4
    push 5
    call sub_12
    push 5
    call sub_48
    add esp, 4
    INVOKE ExitProcess, 0
main endp

sub_12 PROC
    push ebp
    mov ebp, esp
    mov eax, 10
    mul DWORD PTR [ebp+12]
    pop ebp
    ret 8
sub_12 endp

sub_48 PROC
    push ebp
    mov ebp, esp
    mov eax, [ebp+8]
    mul DWORD PTR [ebp+8]
    pop ebp
    ret
sub_48 endp

My original answer is that sub_12 and sub_48 are both CDECL because the Caller is responsible for cleaning up the stack. But now I keep looking at the [mov ebp, esp] instructions and I'm wondering if this is actually an example of an STDCALL.

Does anyone have any hints for me or some extra piece of information that I might seem to be lacking?

Peter Cordes
  • 328,167
  • 45
  • 605
  • 847
user12191253
  • 39
  • 3
  • 1
    Restoring `ebp` clears up the stack space allocated by the function. This must always be done, regardless of calling convention. Cdecl vs. stdcall is about who cleans up the parameters. – fuz Oct 18 '19 at 16:05
  • I see! I think the problem is that I was never lectured on how to identify a parameter – user12191253 Oct 19 '19 at 21:26
  • In both conventions, parameters are pushed by the caller right before the function is called. Thus, they are located on the stack right above the return address. There is no way to identify the number of parameters. – fuz Oct 20 '19 at 00:36

1 Answers1

1

Here's a good discussion on CDECL vs. STDCALL:

Regardless of calling convention, the callee will typically save the current stack pointer to the frame pointer (EBP) so he can push and pull local variables to/from the stack at will.

When he's ready to return, the callee MUST then restore the stack pointer (ESP) in order for the "return" to succeed.

Q: Make sense? Does that answer your question?

ADDITIONAL INFORMATION:

There are two issues: 1) calling the subroutine (this part is "stdcall" vs. "cdecl" (among others - those aren't the only two options), and 2) returning from the subroutine.

The main difference between the CDECL and STDCALL lies in who is responsible for cleaning the stack for local variables upon "return".

The callee ALWAYS restores the stack pointer. That's the only way "return" can work correctly.

For STDCALL, the callee ALSO clears the stack of it's own local variables.

Roughly speaking:

  • STDCALL might use slightly less space, because the "cleanup code" lives in only one place: in the callee. For CDECL calls, the cleanup must be repeated everywhere the subroutine is called. Your example sub_12 is "STDCALL".

  • CDECL is more flexible: it allows you to pass a variable number of parameters into the subroutine. Your example sub_48 is "CDECL".

'Hope that helps...

paulsm4
  • 114,292
  • 17
  • 138
  • 190
  • Thanks for your help! I did read that discussion before posting my own question but it didn't help clear up my misunderstanding. When you say '''the callee MUST restore the stack pointer'' then I think I am more confused. Because I thought the different between STDCALL and CDECL is if the Caller is responsible for the ESP or if the CALEE is responsible for the ESP. If the Callee ALWAYS cleans it up then I'm not sure when CDECL applies. – user12191253 Oct 18 '19 at 15:51
  • 3
    The callee *always* cleans up its own stack frame. With stdcall, the callee *also* cleans up the function parameters that were pushed by the caller. In your example code, this is done by the “ret 8” instruction, which adds 8 to esp after popping the return address. – prl Oct 18 '19 at 16:11
  • 2
    In your example code, the instruction “add esp, 4” (in the calling function) cleans up the parameter that was passed to the cdecl function. – prl Oct 18 '19 at 16:16
  • 2
    *For CDECL calls, the cleanup must be repeated everywhere the subroutine is called* Only if you don't optimize the callers. Instead of `add esp, 8` / `push` / `push`, you could just `mov [esp], eax` / `mov [esp+4], edi` to write new args onto the stack. i.e. leave the arg-passing space allocated and reuse it for future calls in the same function. That's less easy with branching, and can be worse for code-size (in bytes). Of course both conventions are pretty terrible compared to conventions with some register args, like Windows `vectorcall` or x86-64 calling conventions. – Peter Cordes Oct 18 '19 at 18:31
  • @Peter Cordes: My challenge is to balance a short, "meaningful" answer to a complex topic ... without being *too* superficial. I tried to suggest that there are other calling conventions (especially on other platforms). And yes, these conventions became popular with DOS, when every byte mattered, the CPU had a relatively small number of registers, and a 640K address space seemed more than anybody would ever need :( – paulsm4 Oct 18 '19 at 19:31
  • Yup, it's a reasonable oversimplification. My comment is more of a footnote than a correction. – Peter Cordes Oct 18 '19 at 19:33
  • Thank you for your help! This did help clarify somewhat, I guess I was missing the concept that the cleanup happens regardless? Also yes I've heard of the other convention calls but I was trying to focus on these two because I knew they were used more commonly – user12191253 Oct 18 '19 at 20:07
  • So what do you mean by "the cleanup"? One part is to restore the stack pointer. The callee must do this regardless. Another part is to free the storage used for the subroutine's parameters. This second part is where "stdcall" vs. "cdecl" comes into play. Q: Clarify things for you? – paulsm4 Oct 18 '19 at 20:21