ENTER and LEAVE in Assembly?

Question

I was reading The Art of Assembly Language (Randall Hyde, link to Amazon) and I tried out a console application in that book. It was a program that created a new console for itself using Win32 API functions. The program contains a procedure called LENSTR, which stores the length of string in the EBP register. The code for this function is as follows:

LENSTR PROC
ENTER 0, 0
PUSH  EAX
;----------------------
CLD
MOV   EDI, DWORD PTR [EBP+08H]
MOV   EBX, EDI
MOV   ECX, 100 ; Limit the string length
XOR   AL, AL
REPNE SCASB ; Find the 0 character
SUB   EDI, EBX ; String length including 0
MOV   EBX, EDI

DEC   EBX
;----------------------
POP   EAX
LEAVE
RET   4
LENSTR ENDP

Could you explain the usage of the enter and leave commands here?

Answer 1

Enter creates a stack frame, and leave destroys a stack frame. With the 0,0 parameters on the enter, they're basically equivalent to:

; enter
push ebp
mov ebp, esp

; leave
mov esp, ebp
pop ebp

Although it's not used in the code you posted, enter does support doing a bit more than the simple push/mov combination shown above. The first parameter to enter specifies an amount of space to allocate for local variables. For example, enter 5, 0 is roughly equivalent to:

push ebp
mov ebp, esp
sub esp, 5

Enter also supports languages like Pascal that can use nested functions/procedures:

procedure X;
    procedure Y;
    begin
        { ... }
    end
begin
   { ... }
end

In a case like this, Y has access not only to its own local variables, but also to all variables local to X. These can be nested to arbitrary depth, so you could have a Z inside of Y that had access to its own local variables, and the variables of Y and the variables of X. The second parameter to enter specifies the nesting depth, so X would use enter Sx, 0, Y would use enter Sy, 1 and Z would use enter Sz, 2 (where Sx, Sy and Sz signify the size of variables local to X, Y and Z respectively).

This would create a chain of stack frames to give Z access to variables local to Y and X, and so on. This becomes fairly non-trivial if the functions are recursive, so an invocation of Z can't just walk up the stack to the two most recent stack frames--it needs to skip across stack frames from previous invocations of itself, and go directly back to stack frames for the lexical parent function/procedure, which is different from its caller in the case of recursion.

This complexity is also why C and C++ prohibit nested functions. Given the presence of enter/leave, they're fairly easy to support on Intel processors, but can be considerably more difficult on many other processors that lack such direct support.

This also at least helps explain one other...feature of enter--for the trivial case being used here (i.e., enter 0, 0) it's quite a bit slower than the equivalent using push/mov.

Answer 2

This is the setup for the stack frame (activation record) for the function. Internally it normally looks something like this:

    push( ebp );         // Save a copy of the old EBP value
     
    mov( esp, ebp );     // Get ptr to base of activation record into EBP
     
    sub( NumVars, esp ); // Allocate storage for local variables.

// ENTER with a non-zero immediate does all 3 of the above things, slowly.

Then when the stack frame is to be destroyed again, you have to do something along the following lines:

   mov( ebp, esp );    // Deallocate locals and clean up stack.
 
   pop( ebp );         // Restore pointer to caller's activation record.
// LEAVE does the above steps; a RET instruction is separate

   ret();              // Return to the caller.

Here is a better explanation of it using HLA. Though it is well explained in the book you're reading, as I have that book too, and I've read the section explaining it.

Answer 3

Enter and leave just setup the stack frame. Usually compilers generate code that directly manipulates the stack frame pointers as enter and leave aren't exactly fast relative to mov/sub (they used to be though, back in the 286 days :-) ).