push / pop eax gives me "illegal instruction" at compile time

Question

I'm just starting with fasm / x86 programming and i started with the 64 bit samples as my target for my project in which i will need some assembly is 64 bit xeons.

I started from the PE64DEMO and modified it to add a loop, however it fails after the first iteration as (from what i could gather online) the Windows api functions can change registers and not restore them. I also read i'm supposed to save what i need myself pushing and poping from the stack, i commented the push and pop, if i uncomment them i get a compile time error stating "Error : illegal instruction pop eax".

Whole file code bellow :

; Example of 64-bit PE program

format PE64 GUI
include 'win64a.inc'
entry start


section '.text' code readable executable

  start:
    sub     rsp,8*5         ; reserve stack for API use and make stack dqword aligned



    ; Assembly code representing expr1 here (for instance mov [count], 0)
    mov eax,10
    .for_loop_check:
    ; Assembly code representing expr2 that jumps to .exit_for_loop when false (for instance cmp [count], TOP_VALUE / jae .exit_for_loop when expr2 is "count < TOP_VALUE"). Note that if expr2 is absent then so is the jump.

    cmp eax,0
    jz .exit_for_loop

    ;push eax

    ; body code here
    mov     r9d,0
    lea     r8,[_caption]
    lea     rdx,[_message]
    mov     rcx,0
    call    [MessageBoxA]

    ;pop eax

    ; Assembly code representing expr3 here (for instance inc [count])
    dec eax
    jmp .for_loop_check

    .exit_for_loop:

    mov     ecx,eax
    call    [ExitProcess]

section '.data' data readable writeable

  _caption db 'Win64 assembly program',0
  _message db 'Hello World!',0

section '.idata' import data readable writeable

   library kernel32,'KERNEL32.DLL',\
      user32,'USER32.DLL'

  include 'api\kernel32.inc'
  include 'api\user32.inc'

Possible duplicate of [Push and Pop on AMD64](http://stackoverflow.com/questions/5050186/push-and-pop-on-amd64) — Ari0nhh, Jun 07 '16 at 05:53
@Ari0nhh I wouldn't say it's a duplicate (this is useful for someone else searching for fasm and illegal instruction with pop or push, googling didn't yield anything). However it sounds like it could be the same solution, i'll test it and come back here — Ronan Thibaudau, Jun 07 '16 at 09:31
In 64 bit, you can indeed only push and pop 64 bit registers, so it should be `push rax` and `pop rax`, not `eax`. — Rudy Velthuis, Jun 07 '16 at 09:49
@RudyVelthuis I just finished testing based on that, it still doesn't work but it does fix the issue in my question (fasm no longer fails at compile time). If you post this as an answer i'll accept it. However if you have any idea what could also happen for my new problem to avoid a new question that would be great. I replaced eax with rax, everything works if i leave push / pop commented (except of course the loop goes on forever) but as soon as i uncomment them i get an access violation Inside MessageBox, am i not doing the right thing by pushing a register and poping it once win returns? — Ronan Thibaudau, Jun 07 '16 at 09:58
Different platforms, like Windows 64 and Linux 64 bit have an ABI which tells you which registers are used for what when calling a function, and whihc registers must be preserved and which don't. RAX or EAX are often used to hold a return value from the function. — Rudy Velthuis, Jun 07 '16 at 10:30
Thanks for the ABI keyword, helped me find everything i need on msdn! — Ronan Thibaudau, Jun 07 '16 at 11:18
If you found the ABI then you may have realized that you could use a non-volatile register (one that must be preserved) by functions. _RBX_ instead of _RAX_ for a loop counter would work. You may ask why does `PUSH/POP` make it fail. There is a requirement before a call on 64-bit Windows the stack be aligned to a 16-byte boundary. Without the PUSH/POP the stack is 16-byte aligned when calling messagebox etc. Upon entry to start it is misaligned by 8 (ret address on stack). You subtract 40 from stack at start. 48 is evenly divisible by 16. If you add a _PUSH_ the stack is misaligned by 8 again — Michael Petch, Jun 07 '16 at 11:47
Moral of the story: ensure that when you make a call to a function that the stack is aligned to a 16-byte boundary at the point just before the call is made. — Michael Petch, Jun 07 '16 at 11:50
@RonanThibaudau: See also the [x86 tag wiki](http://stackoverflow.com/tags/x86/info) for links to docs and guides. — Peter Cordes, Jun 07 '16 at 18:15
@MichaelPetch Thanks that does help a lot, do you want to turn this into an answer so that i can accept it or should i do it? — Ronan Thibaudau, Jun 08 '16 at 02:06

score 2 · Answer 1 · answered Jun 08 '16 at 02:28

There were multiple issues with my code.

To start with as Rudy Velthuis commented my code was illegal and rightfully rejected by fasm, push eax and pop eax are never valid in X64 as eax is a 32 bit register and X64 doesn't allow pushing 32 bit values to the stack. Switching to a 64 bit register fixed this issue.

;change
push eax
pop eax
;to
push rax;
pop rax;

With that it now compiles and allows me to preserve my register accross the call however the code now segfaults inside the Windows API. As Michael Petch mentioned this is because once i push to the stack i break the alignment that was previously done and Windows 64 requires the stack to be 16 bytes alligned, however instead of fixing the alignment the solution is simplpy to use a non volatile register among those described in the Windows X64 ABI and discard the use of the stack for this operation.

Final code with changes :

format PE64 GUI
include 'win64a.inc'
entry start

section '.text' code readable executable

  start:
        sub     rsp,8*5         ; reserve stack for API use and make stack dqword aligned

        ;use rbx as counter as it is preserved across windows API calls
        mov rbx,10

        .for_loop_check:

        cmp rbx,0
        je .exit_for_loop

        mov     r9d,0
        lea     r8,[_caption]
        lea     rdx,[_message]
        mov     rcx,0
        call    [MessageBoxA]

        ;no more pushing and poping to the stack required, rbx is guaranteed to be unmodified by the windows API

        dec rbx
        jmp .for_loop_check

        .exit_for_loop:

        mov     ecx,eax
        call    [ExitProcess]

section '.data' data readable writeable

  _caption db 'Win64 assembly program',0
  _message db 'Hello World!',0

section '.idata' import data readable writeable

  library kernel32,'KERNEL32.DLL',\
          user32,'USER32.DLL'

  include 'api\kernel32.inc'
  include 'api\user32.inc'

Note that if you were writing a function that returned, instead of calling `ExitProcess`, you'd need to save/restore `rbx` instead of just clobbering your caller's `rbx`. Since you don't need to, it's better not to waste instructions doing that, but that instruction deserves a comment saying why it's safe in this case. — Peter Cordes, Jun 08 '16 at 03:09

push / pop eax gives me "illegal instruction" at compile time

1 Answers1

Linked