3

.

I have a school project, recoding a strace-like command on a x86_64 OpenSUSE. (Intell i7)

For this purpose we are, of course, using ptrace system call but it is forbiden to use PTRACE_SYSCALL. We have to use PTRACE_SINGLESTEP and detect systems calls thanks to PTRACE_PEEKTEXT and opcodes corresponding to system calls instructions (0x80CD for int0x80, 0x050F for syscall and 0x340F for sysenter).

Until there, I'm good. But then we have to fetch the parameters of the system call. For syscall and intx80 it's kind of easy, I look into rax to know which system call it is, then I look into rdi, rsi, rdx, etc.

But for sysenter I cannot find how it's really working. So I tried to code a little assembly program to test those 3 instructions.

    BITS 64

    global main

    section .text

main:
    push rbp
    mov rbp, rsp

    mov rdi, 1
    mov rsi, FormatStr
    mov rdx, 30
    mov rax, 1

    syscall

    leave
    ret

    section .rodata
    FormatStr db 'Hello World ! Sysenter Test !',0Ah,0

Which works perfectly fine !

Now for the int 0x80 version I just change the number of the system call in rax from 1 to 4. (In 32, dunno why but the system calls numbers aren't the same)

    BITS 64

    global main

    section .text

main:
    push rbp
    mov rbp, rsp

    mov rdi, 1
    mov rsi, FormatStr
    mov rdx, 30
    mov rax, 4

    int 0x80

    leave
    ret

    section .rodata
    FormatStr db 'Hello World ! Sysenter Test !',0Ah,0

Which works at 50%. A string is displayed but it's garbage.

Now if I put a sysenter I get a SIGILL signal. I tried with 1 and 4 in rax.

My project just has to run on my computer but I have to be able to detect and analyse binaries who are using sysenter

Can someone give a little explication on those things ?

Thank you !

Ps : sorry for my bad english

Lks
  • 71
  • 1
  • First, you realize that `syscall` and `sysenter` are different things, right? In your question you talk about `sysenter` but show `syscall` in your code. Which is it? – David Hoelzer May 06 '15 at 09:39
  • Yeah of course I know it's different. I did a lot of research on the topic. My point is, it works with syscall but not with sysenter (SIGILL) but in order to understand how to analyse sysenter system calls I have to test it on a little assembly program – Lks May 06 '15 at 09:50
  • If it works with `syscall` that is not what you say in your question. – David Hoelzer May 06 '15 at 09:53
  • Sorry I don't understand – Lks May 06 '15 at 09:55
  • You post code using `syscall` and say it works 50% of the time... in the comments you say, "My point is, it works with syscall but not with sysenter." I did not take that away from the question you have posted. I'm afraid I have to move on from this, but you might be well served to read this: http://wiki.osdev.org/SYSENTER – David Hoelzer May 06 '15 at 09:58
  • 1
    No I said it works 100% with syscall, does not work very well with int 0x80 and doesn't work at all with sysenter – Lks May 06 '15 at 10:03

0 Answers0