2

I know that calling fork() in C will return -1 if there is an error, but I'm wondering what the error return value is when you call sys_fork in assembly.

I might normally assume that it also returns -1 but I've dealt with sys_brk and the raw syscall in assembly returns something different from the C Brk() wrapper.

Does anyone know what the fork error return value would be in assembly?

(I'm doing 64 bit NASM assembly on Linux)

Cody Gray - on strike
  • 239,200
  • 50
  • 490
  • 574
JMcCoy
  • 306
  • 2
  • 6
  • 3
    IIRC, usually a Linux syscall, in ia32 and x86_64 at least, returns negative of an `errno` value on failure (which the glibc wrapper then negates again and stores into `errno`). – Daniel Schepler Jun 28 '17 at 19:01
  • According to the [x86_64](https://github.com/hjl-tools/x86-psABI/wiki/X86-psABI) ABI, the return value is in `EAX` or `RAX`. Also see [Where is the X86-64 ABI documented?](https://stackoverflow.com/q/18133812/608639) – jww Jun 28 '17 at 19:02
  • He knows that the return value is in `RAX` according to the ABI, @jww. That is stated in the question. The question is how to *interpret* the value in `RAX`. In other words, what does it mean? – Cody Gray - on strike Jun 28 '17 at 19:16
  • @Daniel Schepler sounds like you are saying it will return some value below zero upon failure, depending on the type of error that was encountered. So if I don't care exactly what went wrong I can do: "cmp rax, 0 (next line) jl sys_fork_error"? – JMcCoy Jun 28 '17 at 20:57
  • what did it return when you tried it? – old_timer Jun 28 '17 at 21:22
  • 1
    @old_timer just as it does with the C fork() syscall, it returns 0 in the child process and the pid in the parent process. But this is with a successful call. I don't know how I would force a fork() failure and check the return value. – JMcCoy Jun 29 '17 at 02:07

1 Answers1

3

First and foremost note that the C library wrapper fork(2) invokes sys_clone and not sys_fork.

C library/kernel differences

Since version 2.3.3, rather than invoking the kernel's fork() system
call, the glibc fork() wrapper that is provided as part of the NPTL
threading implementation invokes clone(2) with flags that provide the
same effect as the traditional system call.

The introduction to the section 2 of the Linux manual explains how to interpret the return value of a system call in a general context:

RETURN VALUE

On error, most system calls return a negative error number (i.e., the
negated value of one of the constants described in errno(3)). The C
library wrapper hides this detail from the caller: when a system call
returns a negative value, the wrapper copies the absolute value into
the errno variable, and returns -1 as the return value of the
wrapper.

So for most system calls, EAX/RAX holds -ESOMETHING on error, or a non-negative result on success. The libc wrappers decode this to implement the errno-setting and returning -1 behaviour described in the section 2 man pages, which primarily document the wrappers; "C library/kernel differences" details are sometimes found in the Notes section of Linux man pages.

It's important to note that this applies to most but not all system calls, as the first paragraph says. sys_fork is not special in this regard. A couple interesting special cases are getpriority as mentioned in the beginning of errno(3)more below, and mmap. (Valid pointers can have their high bit set, so distinguishing error from success requires other tricks, like checking the low bits since a successful mmap always returns a page-aligned address.) These ABI details are not documented in the man pages.

For the purpose of finding out if the sys_fork call was successful, testing for a negative value is enough:

test eax, eax
jl _error_handler              ;See Peter Cordes's comments below

I've included the part about the C library wrappers like fork(2) because it gives a practical method to find out the error numbers.
The value of errno, negated, are the possible error values the system call can return.

EAGAIN
ENOMEM
ENOSYS
ERESTARTNOINTR

In general, the C library wrapper can add, remove or transcode the return values before writing them into errno, so this is failable.

The definitive way to find out the possible return values is looking at the source.
For example, do_fork, invoked by sys_fork, can return EINVAL and EPERM in addition to the values listed above.
Other values are possible, I haven't dug into all the nested function calls.

sys_clone also invokes do_fork so I would assume that clone(2) can return all of the error numbers that fork(2) can.

Investigating the case of sys_getpriority mentioned above a comment came up

/*
 * Ugh. To avoid negative return values, "getpriority()" will
 * not return the normal nice-value, but a negated value that
 * has been offset by 20 (ie it returns 40..1 instead of -20..19)
 * to stay compatible.
 */

So it seems that Linux system calls always return negative values on an error, the C library wrappers than, in an attempt to normalise these values into errno, introduce an extra layer of complexity.

As we see from the mmap case, having the sign bit set doesn't always mean it's an error value. According to this answer (merit goes to Peter Cordes comment below), values in the range [-4095, -1] always mean error, but other negative values should not.

Margaret Bloom
  • 41,768
  • 5
  • 78
  • 124
  • Are you sure you need to test the full `rax`, and not just `eax`? `pid_t` is a 32-bit type, so I think it's safe to avoid a REX prefix here. (Actually, are you sure the return value is sign-extended out to 64 bits at all?) – Peter Cordes Jul 01 '17 at 14:39
  • @PeterCordes I don't know honestly, I trusted the comments under the question thinking that in the worst the PID should sign-extended like handles are in Windows. I'll correct the answer with your input. Thank you! – Margaret Bloom Jul 01 '17 at 15:37
  • I checked with `sys_open(argv[1], 0 /*O_RDONLY*/)` ([NASM source](https://pastebin.com/hx2n4LSz)) on amd64 Linux 4.11. **Negative syscall return values are indeed sign-extended to the whole RAX**, even though `open(2)` returns an `int`. Still, you might as well just check `eax` if the return value is a signed 32-bit integer. – Peter Cordes Jul 02 '17 at 03:10
  • Your point about "most" but not all got me thinking about mmap. I found [a blog post](http://nullprogram.com/blog/2016/09/23/) with details. `musl` libc apparently uses `if (r > -4096UL)` for *all* system calls to detect errors, unless there are any special cases that don't use the usual helper function to decode such return values into errno and return `-1`. I made an edit to your answer, but you might want to revise it some more. – Peter Cordes Jul 02 '17 at 04:26
  • 1
    See also https://stackoverflow.com/questions/18996410/linux-kernel-system-call-returns-1-instead-of-1-256/18998521#18998521. Apparently values from -1 to -4095 always mean error, and other values always mean non-error. I guess that's why `mmap` doesn't mention decoding the return value as a special case, but `getpriority` needs to encode the return value with an offset. (I wish I'd found that before I spent time on an edit. I don't really want to do another edit, but maybe you'll want to.) – Peter Cordes Jul 02 '17 at 04:31
  • I knew this issue looked familiar. Turns out I mentioned it [in an answer last year](https://stackoverflow.com/questions/38751614/what-are-the-return-values-of-system-calls-in-assembly/38752895#38752895). I just fixed it to link here for more details of decoding error values. – Peter Cordes Jul 02 '17 at 04:47
  • Thank you very much @Peter, I'll look into that! – Margaret Bloom Jul 02 '17 at 11:11