0

I'm implementing my own kernel. I have a problem when I try to use syscall instruction. I already defined the setup for syscall, but when i call syscall command the 'INVALID OPCODE' (Interrupt 6 (#UD)) interruption is called.

I don't know what is happening to cause this exception. The code that invoke the syscall is the last to execute, if i comment the last line when i call 'syscall' no exception is called.

My syscall setup:

setup_syscall:
    ; Set (0xC0000081) bits 32-47 are kernel segment base, bits 48-63 are user segment base (ON GDT)
    mov ecx, 0xC0000081
    rdmsr
    
    ; setup with kernel segment inside and outside syscall
    mov edx, 0x0808
    wrmsr

    ; (0xC0000082) The kernel's RIP SYSCALL entry for 64 bit software
    mov ecx, 0xC0000082
    mov rdx, syscall_handler
    mov eax,edx
    shr rdx,32
    wrmsr

    
    ; Set IA32_EFER.SCE (0xC0000080) and SCE (SYSCALL Enable) is its 0th bit
    mov ecx,0xC0000080 
    xor rax,rax
    rdmsr
    bts rax,1
    wrmsr

    ret

My syscall invoke:

test_syscall:
    ; Enable and setup syscall 
    call setup_syscall
    ; test syscall
    syscall

    jmp $

My syscall:

section .text
[bits 64]

[global syscall_handler]

syscall_handler:
    jmp $   ; infinite loop
    sysret

My GDT:

ALIGN 4
GDT:
.Null:
    ; Null Descriptor - should be present.
    dq 0
.Code: equ $ - GDT
    ; 64-bit code descriptor (exec/read).
    dq (1<<43)|(1<<44)|(1<<47)|(1<<53)|(1<<41)
.Data:
    ; 64-bit code descriptor (exec/read).
    dq (1<<44)|(1<<47)|(1<<41)
.Pointer:
    dw $ - GDT - 1                    ; 16-bit Size (Limit) of GDT.
    dd GDT                            ; 32-bit Base Address of GDT. (CPU will zero extend to 64-bit)

Infos: Until now, my OS is in Long mode, has mapped the avaliable memory, but dont have any setup for ring 3, so all code is executed in ring 0, because of this, the base segments I use in the syscall setup is 0x8 for kernel and user.

Crazynds
  • 61
  • 1
  • 9
  • Is "user-space" also in full 64-bit long mode? If not, if you're using 32-bit compat sub-mode of long mode, then `syscall` is an invalid instruction on Intel CPUs, only works in 32-bit mode on AMD CPUs. Note Intel's documentation: https://www.felixcloutier.com/x86/syscall – Peter Cordes May 29 '23 at 17:29
  • @PeterCordes Hi, thanks for the commentary. My kernel don't have user space yet, so all the code is running in kernel space, because of this is set up my segments for syscall to 0x8. Despite that, I'm suspecting that it could be something related to what you commented, because according to the documentation you sent me, the processor just doesn't recognize the command if it's not in long mode. Is there any way to test the way my execution is? – Crazynds May 29 '23 at 17:55
  • Do you run it on real hardware, if yes, what cpu. If not, what VM :) – Jester May 29 '23 at 18:09
  • Bochs' built-in debugger knows what mode the machine is in, so if it's disassembling 64-bit instructions like `mov rdx, syscall_handler` as expected, then you know you're in 64-bit mode. (CS.L = 1). It should also tell you what caused a fault. – Peter Cordes May 29 '23 at 18:10
  • @Jester I'm using qemu to emulate a x86_64 hardware. – Crazynds May 29 '23 at 18:18
  • @PeterCordes I never used bochs, but i will try use this emulator to find what happened. – Crazynds May 29 '23 at 18:19
  • 3
    `bts rax, 1` should be `bts rax, 0` or `or rax, 1` – Jester May 29 '23 at 18:26
  • @Jester Thanks for the commentary. This really resolves my problem, but when I call sysret the emulator restart, not calling any exception. My syscall handler is only the sysret instruction even with `o64` prefix. – Crazynds May 29 '23 at 19:12
  • 1
    You should nevertheless get an exception and probably a triple fault. Anyway, `sysret` is hardwired to return to user mode (ring 3) so you might just want to set that up first. – Jester May 29 '23 at 22:53
  • 1
    That said, it works here. You did not post [mcve] so that's all I can say. – Jester May 29 '23 at 23:12
  • 1
    Another mistake: `mov edx, 0x0808` should be `mov edx, 0x080008`. Even so, returning to 64 bit will use `IA32_STAR[63:48]+16` which does not match your setup (but isn't directly fatal since selectors are not used). – Jester May 29 '23 at 23:23

0 Answers0