Using r8 register as a loop counter results in endless loop - why?

Question

The following code prints hello world 10 times by using the rsi register as a loop counter.

section .data
    hello:     db 'Hello world!',10   
    helloLen:  equ $-hello             

section .text
    global _start

_start:
    mov rsi, 0                 ;<--- use r8 here

do_loop:
    inc rsi                    ;<--- use r8 here

    ;print hello world
    mov eax,4             
    mov ebx,1            
    mov ecx,hello       
    mov edx,helloLen     

    int 80h              

    cmp rsi, 10                ;<--- use r8 here
    jnz do_loop

    ;system exit
    mov eax,1            ; The system call for exit (sys_exit)
    mov ebx,0            ; Exit with return code of 0 (no error)
    int 80h;

If I am trying to use the r8 register instead of rsi as a loop counter it results in an endless loop. The r8 register here is just an example. It happens for the registers r9, r10 as well.

Could someone explain this, because I thought these are all general purpose registers and you should be allowed to use them?

You are using the 32 bit system calls in 64 bit mode. Bad idea. — Jester, Jul 21 '16 at 13:11
Interesting. Maybe Linux's `int 0x80` 32-bit ABI clobbers upper registers when called from 64-bit code? In Linux, `syscall` clobbers rcx and r11, but preserves everything else (except rax for the return value, of course). Usually people run into problems with the `int 0x80` ABI when trying to pass pointers that don't fit in 32 bits. (Resulting in `-EFAULT`) — Peter Cordes, Jul 21 '16 at 13:44
Try single-stepping in gdb with `layout reg` to highlight regs that change. (See the bottom of the [x86 tag wiki](http://stackoverflow.com/tags/x86/info) — Peter Cordes, Jul 21 '16 at 13:46
okay, changing the block which prints the string "hello world" to a 64 bit system call resolves the issue. There must be a weird side effect in this block. — Dennis, Jul 21 '16 at 13:54
Not only the upper bits, but the 64 bit extra registers as a whole are not saved by the 32 bit syscall for obvious reasons. Hardly "weird". — Jester, Jul 21 '16 at 15:05
@Jester: If you don't know how the kernel side of things works, guessing that an interrupt handler saves/restores the full x86-64 state when coming from long-mode user-space is not a bad guess. But I assume the handler for interrupt vector `0x80` is different from the handlers for other interrupts, and uses its own save/restore code. (Also, what's a good term for r8-r15? I said "upper registers", but I guess that sounds like the upper halves of rax-rsp. AMD considered calling them UAX/UCX etc. instead of r8/r9 etc. "extra 64bit registers" seems clumsy :/). — Peter Cordes, Jul 21 '16 at 23:30
Well, the situation seems to be different than I thought, the [compatibility int 80 handler](https://github.com/torvalds/linux/blob/v4.6/arch/x86/entry/entry_64_compat.S#L313-L329) actually explicitly zeroes registers `r8` through `r11` and preserves the full 64 bits of the others (except for `rax` which is of course the return value). — Jester, Jul 22 '16 at 00:22

Michael Petch · Accepted Answer · 2016-07-23T14:14:43.953

TL;DR : int 0x80 implicitly zeroes out R8, R9, R10, and R11 on 64-bit Linux systems prior to returning to userland code. This behavior occurs on kernels later than 2.6.32-rc1. This is not the case for the preferred 64-bit SYSCALL calling convention.

You are experiencing a peculiarity of the Linux Kernels after version 2.6.32-rc1. For Linux kernel versions <= 2.6.32-rc1 you may get the behaviour you were expecting. Due to an information leakage bug (and exploits) the registers R8, R9, R10, and R11 are now zeroed out when the kernel returns from an int 0x80.

You may believe that these registers shouldn't matter in compatibility mode (32-bit code) since those newer registers are unavailable. This is a false assumption as it's possible for a 32-bit application to switch into 64-bit long mode and access those registers. The Linux Kernel Mailing List post that identified this issue had this to say:

x86: Don't leak 64-bit kernel register values to 32-bit processes

While 32-bit processes can't directly access R8...R15, they can gain access to these registers by temporarily switching themselves into 64-bit mode.

Code that demonstrates register leakage on the earlier kernels was made available by Jon Oberheide. It creates a 32-bit application to be run on an x86-64 system with IA32 compatibility enabled. The program switches to 64-bit long mode and then stores registers R8-R11 into general purpose registers that are available in compatibility mode (32-bit mode). John discusses the specifics in this article. He sums the vulnerability and the kernel fix nicely in this excerpt:

The Vulnerability

The underlying issue that causes this vulnerability is a lack of zeroing out several of the x86-64 registers upon returning from a syscall. A 32-bit application may be able to switch to 64-bit mode and access the r8, r9, r10, and r11 registers to leak their previous values. This issue was discovered by Jan Beulich and patched on October 1st. The fix is obviously to zero out these registers to avoid leaking any information to userspace.

If you were to step through your code in a debugger like GDB you should discover that R8 is in fact set to zero after int 0x80. Since it is your loop counter your program ends up in an endless cycle printing Hello world!.

Using r8 register as a loop counter results in endless loop - why?

1 Answers1

Linked