Prologue of "add esp, 0FFFFFFF8h"

Question

I'm a tad bit rusty on my MASM, so I don't really recall what to do here (if anything needs to be done at all). I have a MASM (X86) routine that looks as follows. It has two local variables taking up 5 bytes total:

MSC_ASM_GenerateBlock PROC buffer:DWORD,bsize:DWORD,safety:DWORD
  LOCAL val:DWORD, rc:BYTE  ;; local variables
  MWSIZE EQU 4              ;; machine word size

  .WHILE bsize >= MWSIZE && safety > 0
     ;; RDRAND is not available prior to VS2012. Just emit
     ;;   the byte codes using DB. This is `rdrand eax`.
     DB 0Fh, 0C7h, 0F0h
     setc rc
     ...
  .ENDW
  ...
MSC_ASM_GenerateBlock ENDP

When I check the disassembly, I see:

> dumpbin.exe /DISASM rdrand-x86.obj
Dump of file rdrand-x86.obj

_MSC_ASM_GenerateBlock:
  00000000: 55                 push        ebp
  00000001: 8B EC              mov         ebp,esp
  00000003: 83 C4 F8           add         esp,0FFFFFFF8h
  00000006: EB 1D              jmp         00000025
  00000008: 0F C7 F0           rdrand      eax
  0000000B: 0F 92 45 FB        setb        byte ptr [ebp-5]
  0000000F: 80 7D FB 00        cmp         byte ptr [ebp-5],0
  ...

I believe add esp, 0FFFFFFF8h is another way to say sub esp, 08h.

As Joshua pointed out, the difference between an add esp and sub esp is the flags after the operation. The assembler's confusion or selection of instructions could be based on the fact that the assembler does not get to see the RDRAND context. Rather, it only sees the jmp based on CY, and the assembler believes the flags are not in a good state.

Why is MASM generating a non-intuitive add that depends on unsigned integer wrap? And more importantly, is it OK?

I performed a port of the routing to MASM64/ML64 using the wider machine words. It produces the same code (modulo machine word size):

Dump of file rdrand-x64.obj

MSC_ASM_GenerateBlock:
  0000000000000000: 55                 push        rbp
  0000000000000001: 48 8B EC           mov         rbp,rsp
  0000000000000004: 48 83 C4 F0        add         rsp,0FFFFFFFFFFFFFFF0h
  0000000000000008: EB 1D              jmp         0000000000000037
  0000000000000010: 48 0F C7 F0        rdrand      rax
  000000000000001D: 0F 92 45 F7        setb        byte ptr [rbp-9]
  0000000000000024: 80 7D F7 00        cmp         byte ptr [rbp-9],0
  ...

score 7 · Accepted Answer · answered Oct 14 '15 at 03:47

7

Odd compiler. Very odd.

add esp, 0FFFFFFF8h

is exactly the same as

sub esp, 8h

except it sets the flag bits differently. It's fine, and yes it depends on unsigned integer wrap. Not a problem because assembly is inherently non-portable. If you wanna know why you'll have to ask Microsoft, and they probably don't know anymore.

answered Oct 14 '15 at 03:47

Joshua

40,822
8
72
132

*"...except it sets the flag bits differently"* - maybe that's it. Though `RDRAND` sets the flags, the compiler cannot tell because this is targeting `cl.exe` and `ml.exe` shipped with VS2005, VS2008 and VS2010 (none of them can emit `RDRAND`). All the compiler and assembler see/understand is a jump based on the `CY`, without the `RDRAND` context. – jww Oct 14 '15 at 04:05
1

But it isn't it. You don't check flag bits after changing esp. – Joshua Oct 14 '15 at 15:08
2

@jww One possibility for why the compiler does this might be related to the situation where it needs to allocate exactly 128 bytes for local stack space. There is a difference between `add` and `sub` in that case. `add esp, -128` can still be encoded in 3 bytes but `sub esp, 128` would take 6 bytes. With only 8 bytes needed in this case it doesn't matter (both `add` and `sub` encode to 3 bytes). – Michael Petch Dec 30 '16 at 07:48

score 0 · Answer 2 · edited May 23 '17 at 12:33

0

I have no idea why MASM would think that's a good idea but unsigned overflow will work (so this is safe), and even more confusingly this is a safe substitution to make even on a 64 bit processor because the 32 bit operand will zero out the high 32 bits of the register! See this for an explanation.

Also, since both instructions are 3 bytes long there isn't any inefficiency there.

edited May 23 '17 at 12:33

Community

1
1

answered Oct 14 '15 at 04:11

dave

4,812
4
25
38

1

Well, except in 64-bit mode the high 32 bits of RSP might not be 0. – Ross Ridge Oct 14 '15 at 05:11
They will be after a 32 bit instruction! – dave Oct 14 '15 at 05:18
Which means it's not safe in 64-bit mode. – Ross Ridge Oct 14 '15 at 05:20
@RossRidge: yes it is, the top 32 bits will be 0 whether you do a 32 bit add or a 32 bit subtract. And therefore it is fine to substitute that instruction. – dave Oct 14 '15 at 05:21
1

@RossRidge: you clearly don't understand what is being asked and answered here ... the question is: "can the 32 bit sub instruction be replaced with this 32 bit add instruction" and the answer is yes because the top 32 bits will be 0 after _either_ instruction is executed. That a 32 bit sub instruction might not be safe in 64 bit mode is COMPLETELY irrelevant. – dave Oct 14 '15 at 05:24
No, you clearly don't understand your own answer. You said it was safe in 64-bit mode. You're wrong. – Ross Ridge Oct 14 '15 at 05:47
It is safe to replace one 32 bit instruction with the other 32 bit instruction because both of them do the same thing. That's what I mean by safe (i.e. the two instructions do the same thing EVEN in 64 bit mode). Your assertion that that isn't the case is WRONG. – dave Oct 14 '15 at 05:48
The fact that `sub esp, 8` is unsafe in 64-bit mode doesn't make your claim that `add esp,0FFFFFFF8h` is safe in 64-bit mode any less wrong. – Ross Ridge Oct 14 '15 at 05:52
Ahhh, you are almost there .... my assert is that it is safe to replace `sub esp,8` with `add esp,0FFFFFFF8h` and that is surprisingly true even in 64 bit mode (where one would expect that the add wouldn't overflow) but it is safe to do that instruction change. – dave Oct 14 '15 at 05:56
Except that's not what you wrote in your answer, and it still wrongly implies that `add esp,0FFFFFFF8h` would somehow be safe in 64-bit mode. – Ross Ridge Oct 14 '15 at 06:03
The question asks "can instruction x be replaced by instruction y which overflows". And my answer says yes it can and it can do this even in 64 bit mode. You have to read the question as well as my answer in order to understand it. Your assertion is that you could never want to run a 32 bit instruction in 64 bit mode. And somehow you think that `add esp, 0FFFFFFF8h` doesn't overflow in 64 bit mode (and thus give the same result as subtracting 8). All your comments thus are confusing to anyone reading it. – dave Oct 14 '15 at 06:15
Now you're just making things up. I never said either of those things. If you don't want to fix your post, that's up to you. There's no need to lie about what I wrote. You're not fooling anyone but yourself. – Ross Ridge Oct 14 '15 at 06:44
I'm confused about what you think needs to be fixed in my post. I still don't see it. What on earth is wrong with replacing `sub esp, 8` with `add esp, 0FFFFFFF8h` in 64 bit code? – dave Oct 14 '15 at 06:47
The problem is that your answer says that "this is safe even in 64 bit mode". You didn't say "replacing `sub esp, 8` with add `esp, 0FFFFFFF8h` doesn't change the behaviour even 64-bit mode". These two instructions are equivalent in both 32-bit and 64-bit modes, but they're only safe in 32-bit code. In 64-bit mode they're unsafe because they both clear upper 32-bits of RSP. Replacing one unsafe instruction with another unsafe instruction that does the exact same thing doesn't make "this safe even 64 bit mode". – Ross Ridge Oct 14 '15 at 07:19
Oh wow, okay. I didn't see that you had misunderstood for all this time. I'd answered the question which was yes this is a safe substitution even in 64 bit code. Somehow you've read that to mean you could always do this even if esp contained a 64 bit value (which is why I was saying you were claiming that you could never want to run a 32 bit instruction in 64 bit mode). I can make that clearer since it obviously confused you. – dave Oct 14 '15 at 07:26
No one asked if it was "a safe substitution in 64 bit code", that was your "COMPLETELY irrelevant" digression , and it's not a safe substitution since it's not safe either way. In 64-bit code clearing the upper 32-bits of RSP isn't safe. You can't assume the bits are already zero. – Ross Ridge Oct 14 '15 at 07:39
But that makes the substitution safe. If you assume that one instruction is safe (and there's no reason that it has to be unsafe, we could have just loaded a 32 bit number for instance) then the substitution is safe. I can't believe you are still confused. – dave Oct 14 '15 at 12:09
No, it couldn't have just loaded a 32-bit number. The question is about a specific code sequence emitted by MASM as part of a function prologue. It's not safe to use `add esp, 0FFFFFFF8h` in the 64-bit version of this code. You've invented your own "COMPLETELY irrelevent" question that only misleads people into thinking that this instruction would be safe to use in a 64-bit function prologue in order to allocate 8 bytes on stack. – Ross Ridge Oct 14 '15 at 16:07
Ross and Dave - I added the MASM64/ML64 port/disassembly if you would like to see what is produced in this case. – jww Oct 14 '15 at 17:39

Prologue of "add esp, 0FFFFFFF8h"

2 Answers2