NASM x86_64: File opening (SYS_OPEN) error list?

Question

I'm coding a linux x64 assembly program that read a file and I want to handle errors like File Not Found or permission errors.

Where can I find a list of SYS_OPEN error codes?

Approaches to find codes (kinda fun)

My code to open a file:

SYS_OPEN    equ 2
O_RDONLY    equ 0

section .data
    filename db "file.txt", 0

section .text
    global _start

_start:
    mov rax, SYS_OPEN
    mov rdi, filename
    mov rsi, O_RDONLY
    mov rdx, 0644o
    syscall

    [...]

When the file is successfully opened the RAX register points to the file descriptor (positive integer), if fails RAX point to an error (negative integer). I managed to raise a permission error by removing all permissions for all users:

chmod 0000 file.txt

This causes an error with code -13. By deleting the file, I managed to get error -2. Where can I find a list of SYS_OPEN error codes?

PS: Maybe my googling skills are rusty

Answer 1

You can interpret this as values of errno, here is the table (to list all of the codes use errno -l), also take a look at the docs. A part of the table:

number	hex	symbol	description
2	0x02	ENOENT	No such file or directory
13	0x0d	EACCES	Permission denied

There is described a reason of such decision: https://stackoverflow.com/a/6008711/9926721

Answer 2

Linux system call return values from -4095 to -1 are -errno codes. (The actual highest error number that Linux has actually defined is currently about 133, EHWPOISON, but that's the official range.)

strace ./myprog can decode them for you so you don't need to actually write error checking in your toy programs when playing around with system calls.

For example:

$ strace touch /tmp/xyjklj/bar
...  (dynamic linker / process startup stuff)

openat(AT_FDCWD, "/tmp/xyjklj/bar", O_WRONLY|O_CREAT|O_NOCTTY|O_NONBLOCK, 0666) = -1 ENOENT (No such file or directory)
utimensat(AT_FDCWD, "/tmp/xyjklj/bar", NULL, 0) = -1 ENOENT (No such file or directory)

... (more system calls as touch(1) finds a locale-specific set of error messages and prints

(The -1 is what the libc wrapper function actually returns; the errno code is what strace decoded from the asm syscall return value, which the glibc wrapper will store in errno. When using raw system calls in asm, you don't have to waste instructions doing that. But strace will still say "-1", not the numeric error code)

---

Documentation of most ways SYS_open can fail

Each system call man page documents which error codes that particular system call can fail with, and in which cases that can happen. (Those list aren't fully exhaustive, for example not covering weird things a specific filesystem like NFS could return, like EMULTIHOP (see comments).)

For your case, see the ERRORs section of the open(2) man page. e.g. there are several entries for ENOENT, covering all the cases which can lead to that return value.

• ENOENT - O_CREAT is not set and the named file does not exist.

• ENOENT - A directory component in pathname does not exist or is a dangling symbolic link.

• ENOENT - pathname refers to a nonexistent directory, O_TMPFILE and one of O_WRONLY or O_RDWR were specified in flags, but this kernel version does not provide the O_TMPFILE functionality.

(Spoiler alert, 2 is ENOENT, so -2 is -ENOENT.)

There are of course lots of other fun ways that pathname and file access stuff (and open(2) in particular) can error, including:

• EACCES (-13) - The requested access to the file is not allowed, or search permission is denied for one of the directories in the path prefix of pathname, or the file did not exist yet and write access to the parent directory is not allowed. (See also path_resolution(7).)

• EFAULT - pathname points outside your accessible address space.

• ENAMETOOLONG - pathname was too long.

• EBUSY - O_EXCL was specified in flags and pathname refers to a block device that is in use by the system (e.g., it is mounted).
  [this would require root, otherwise you'd get EACCESS]

• ETXTBSY - pathname refers to an executable image which is currently being executed and write access was requested.

• EWOULDBLOCK - The O_NONBLOCK flag was specified, and an incompatible lease was held on the file (see fcntl(2)).

• ENODEV - pathname refers to a device special file and no corresponding device exists. (This is a Linux kernel bug; in this situation ENXIO must be returned.)

• ELOOP - Too many symbolic links were encountered in resolving pathname. EISDIR - pathname refers to a directory and the access requested involved writing (that is, O_WRONLY or O_RDWR is set). ENOTDIR - A component used as a directory in pathname is not, in fact, a directory, or O_DIRECTORY was specified and pathname was not a directory.

• EPERM - The O_NOATIME flag was specified, but the effective user ID of the caller did not match the owner of the file and the caller was not privileged.

As well as various limits like number of open files (ENFILE, EMFILE), or ENOSPC disk space full. The above is not a complete list, I just took one each the ways to get many (but not all) of the error codes.

---

As per funnydman's answer, you can look up the number -> symbolic meaning of error values in man pages. Or look in /usr/include/asm-generic/errno-base.h (The full path may differ on some systems, and you'd only include this file indirectly, via #include <errno.h>)