Linux syscalls and errno

Question

Context: I am trying to write a small C program with inline asm that should run under Linux on an x86_64 system and being compiled with gcc in order to better understand how syscalls work under Linux.

My question is: How are error numbers returned from a syscall (e.g. write) in this environment? I understand that when I use a library such as glibc, it takes care of saving the resulting error code in the global errno variable. But where is the error number stored when I call a syscall directly through inline assembler? Will it be stored inside a separate register, or will it be encoded in %rax?

Let take the write syscall on linux as an example:

When call write then after the syscall returns I find it stores 0xfffffffffffffff2 inside %rax, do I need to somehow extract the error code from that?

If I have the error code number, where should I look to identify the actual error that occured? Lets say I get the number 5 returned, which header file do I need to consult to find the corresponding symbolic error name.

I am calling the write syscall like this:

asm ("mov $1,%%rax;"
     "mov $1,%%rdi;"
     "mov %1,%%rsi;"
     "mov %2,%%rdx;"
     "syscall;"
     "mov %%rax,%0;"
     : "=r" (result)
     : "r" (msg), "r" (len)
     : "%rdx", "%rsi", "%rax", "%rdi" /* EDIT: this is needed or else the registers will be overwritten */
    );

with result, msg and len defined like so:

long result = 0;
char* msg = "Hello World\n";
long len = 12;

Answer 1

The Linux syscall's convention is that they encode both the possible error code and the return value for successful call in the return value. It's just glibc or other C libraries's wrappers that they will set errno to the error code returned by the underlying syscall, and the wrapper will return -1. Taking the write as an example, the kernel does the error processing similar to this:

ssize_t write(int fd, ...) {
    if (fd is not valid)
         return -EBADF;
    return do_write(...);
}

So as you can see, the error code is just in the return value, and depending on the semantics, there is always a way to check if the syscall succeeded or not by comparing it to a value not possible for successful operation. For most syscalls, like write, that means check if it is negative.

Answer 2

Architecture Calling Conventions

As you already guessed, you can't use errno because it's GLibC specific. The information you want will be in rax if it's a x86_64. The man page man 2 syscall has the following explanation:

Architecture calling conventions

       Every architecture has its own way of invoking and passing arguments
       to the kernel.  The details for various architectures are listed in
       the two tables below.

       The first table lists the instruction used to transition to kernel
       mode (which might not be the fastest or best way to transition to the
       kernel, so you might have to refer to vdso(7)), the register used to
       indicate the system call number, the register used to return the
       system call result, and the register used to signal an error.

       arch/ABI    instruction           syscall #  retval  error    Notes
       ────────────────────────────────────────────────────────────────────
       alpha       callsys               v0         a0      a3       [1]
       arc         trap0                 r8         r0      -
       arm/OABI    swi NR                -          a1      -        [2]
       arm/EABI    swi 0x0               r7         r0      -
       arm64       svc #0                x8         x0      -
       blackfin    excpt 0x0             P0         R0      -
       i386        int $0x80             eax        eax     -
       ia64        break 0x100000        r15        r8      r10      [1]
       m68k        trap #0               d0         d0      -
       microblaze  brki r14,8            r12        r3      -
       mips        syscall               v0         v0      a3       [1]
       nios2       trap                  r2         r2      r7
       parisc      ble 0x100(%sr2, %r0)  r20        r28     -
       powerpc     sc                    r0         r3      r0       [1]
       s390        svc 0                 r1         r2      -        [3]
       s390x       svc 0                 r1         r2      -        [3]
       superh      trap #0x17            r3         r0      -        [4]
       sparc/32    t 0x10                g1         o0      psr/csr  [1]
       sparc/64    t 0x6d                g1         o0      psr/csr  [1]
       tile        swint1                R10        R00     R01      [1]
       x86_64      syscall               rax        rax     -        [5]
       x32         syscall               rax        rax     -        [5]
       xtensa      syscall               a2         a2      -

And note number [5]:

[5] The x32 ABI uses the same instruction as the x86_64 ABI and
               is used on the same processors.  To differentiate between
               them, the bit mask __X32_SYSCALL_BIT is bitwise-ORed into the
               system call number for system calls under the x32 ABI.  Both
               system call tables are available though, so setting the bit
               is not a hard requirement.

(In that man page, a table showing how to pass arguments to system calls follows. It's an interesting read.)

---

How are error numbers returned from a syscall (e.g. write) in this environment?:

You gotta check your rax register for the return value.

Answer 3

On Linux, a failed system call using the syscall assembly instruction will return the value -errno in the rax register. So in your case 0-0xfffffffffffffff2 == 0xE which is 14. So your errno is 14.

How do you find what errno 14 means? You should google search "Linux error code table" or look in errno.h and you'll find the answer.

Take a look here: http://www.virtsync.com/c-error-codes-include-errno

According to that table, 14 is EFAULT which means "Bad address".

Answer 4

IIRC in the x86-64 ABI, an error is transmitted from the syscall with the carry bit set. Then eax contains the errno code.

I would suggest to study the lower layers of the source code of some libc library, like musl-libc