Unix O_CREAT flag without mode specified

Question

The definition of the UNIX open() function when used with the O_CREAT flag is that it requires a third argument named mode in order to set the files' privileges.

What if that mode is not specified?

int file;
static const char filename[] = "test.test";

if ((file = open(filename, O_RDWR | O_CREAT | O_TRUNC)) == 1)
{
    perror("Error opening file.");
    exit(EXIT_FAILURE);
}

close(file);

What happens with the file that is created using those flags? On my system I get:

-r--r-s---  1 hyperboreean hyperboreean     0 2009-02-25 01:40 test.test

A theory is that the open function looks on the stack and checks for the mode parameter and ends up using a random integer it finds.

What does the standard say about this?

Answer 1

The POSIX standard (IEEE 1003.1:2008) prototypes open() as:

int open(const char *path, int oflag, ...);

The section describing the behaviour of O_CREAT doesn't say what will happen if you omit the necessary third argument, which means the behaviour is undefined - anything is possible.

In practice, the use of part of the stack that was intended to be stack frame or return address or something similar is quite likely - unto a reasonable approximation, that can be considered a random integer.

The POSIX 2008 standard has some interesting new (and useful) flags for open(), including:

• O_FDCLOEXEC to specify close-on-exec at open.
• O_DIRECTORY to specify that the file must be a directory.
• O_NOFOLLOW to specify not to chase symlinks.

Answer 2

Good question. The mode value will be modified by the umask of the process. So if you don't pass a mode explicitly to open in an O_CREAT operation, and if this results in random bits being used for the mode, those random bits will be modified by the umask.

Wish I could be more definitive and precise, but I agree with cdonner that "random" values are being used, along with the umask.

Edit: One thing you could try is to use dtruss or truss or some other facility to trace system calls, and look at the value of mode at run-time to see if something sensible is used, or if it's just random bits modified by the umask, for example.

Answer 3

For the record, on most libc systems, you will probably be on hands of va_arg, which states in it's man page:

   If there is no next argument, or if type is not compatible with the
   type of the actual next argument (as promoted according to the
   default argument promotions), **random errors will occur**.

int
__libc_open64 (const char *file, int oflag, ...)
{
    int mode = 0;

    if (oflag & O_CREAT)
    {
        va_list arg;
        va_start (arg, oflag);
        mode = va_arg (arg, int);
        va_end (arg);
    }

    if (SINGLE_THREAD_P)
        return INLINE_SYSCALL (open, 3, file, oflag | O_LARGEFILE, mode);

    int oldtype = LIBC_CANCEL_ASYNC ();

    int result = INLINE_SYSCALL (open, 3, file, oflag | O_LARGEFILE, mode);

    LIBC_CANCEL_RESET (oldtype);

    return result;
}

Answer 4

hyperboreean, your suspicion may be not so far off the mark. I was hoping to find the answer in Kernighan Ritchie. Unfortunately, I did not. I think the permissions parameter is required with the O_CREAT flag, and if you don't provide it, open() will pull a random value from the stack, which of course goes unnoticed in C.

Edit: by "random" I mean not predictable. It probably is picking up part of the return address, which sits on top of the parameters on the stack.

Answer 5

   int open(const char *pathname, int flags);
   int open(const char *pathname, int flags, mode_t mode);

...open()... O_CREAT flag...
What if that mode is not specified?

In addition to other people's answers, if you want to future-proof against this by getting a compiler error when you forgot to specify the mode flag in those cases where it is needed(ie. O_CREAT or O_TMPFILE, according to man 2 open), you'd have to use the GNU project C and C++ compiler (eg. the gcc command) with arg. -D_FORTIFY_SOURCE=1 (or 2) and an optimization flag eg. -O1 or the usual -O2 (because _FORTIFY_SOURCE requires compiling with optimization (-O)).

For example:

#include <stdio.h>
#include <stdlib.h>
#include <fcntl.h>
#include <unistd.h>

int main() {
int file;
static const char filename[] = "test.test";

if ((file = open(filename, O_RDWR | O_CREAT | O_TRUNC)) == 1)
{
    perror("Error opening file.");
    exit(EXIT_FAILURE);
}

close(file);
}

(save that as file: a.c)

$ gcc -D_FORTIFY_SOURCE=2 -O1 a.c
In file included from /usr/include/fcntl.h:328,
                 from a.c:3:
In function ‘open’,
    inlined from ‘main’ at a.c:10:13:
/usr/include/bits/fcntl2.h:50:4: error: call to ‘__open_missing_mode’ declared with attribute error: open with O_CREAT or O_TMPFILE in second argument needs 3 arguments
    __open_missing_mode ();
    ^~~~~~~~~~~~~~~~~~~~~~

So you get: open with O_CREAT or O_TMPFILE in second argument needs 3 arguments

Pedantics follow:
Note that: -O0 or no -O arg. won't work(ie. it won't tell you that you forgot to add mode because it's as if _FORTIFY_SOURCE was unspecified or simply ignored):

$ gcc -D_FORTIFY_SOURCE=2 -O0 a.c
In file included from /usr/include/bits/libc-header-start.h:33,
                 from /usr/include/stdio.h:27,
                 from a.c:1:
/usr/include/features.h:382:4: warning: #warning _FORTIFY_SOURCE requires compiling with optimization (-O) [-Wcpp]
 #  warning _FORTIFY_SOURCE requires compiling with optimization (-O)
    ^~~~~~~

Using _FORTIFY_SOURCE will protect you like that for other cases too, and for open() there is one more case: open can be called either with 2 or 3 arguments, not more, seen in file /usr/include/bits/fcntl2.h as:

__errordecl (__open_too_many_args,
         "open can be called either with 2 or 3 arguments, not more");
__errordecl (__open_missing_mode,
         "open with O_CREAT or O_TMPFILE in second argument needs 3 arguments");

__fortify_function int
open (const char *__path, int __oflag, ...)
{
  if (__va_arg_pack_len () > 1)
    __open_too_many_args ();

  if (__builtin_constant_p (__oflag))
    {
      if (__OPEN_NEEDS_MODE (__oflag) && __va_arg_pack_len () < 1)
    {
      __open_missing_mode ();
      return __open_2 (__path, __oflag);
    }
      return __open_alias (__path, __oflag, __va_arg_pack ());
    }

  if (__va_arg_pack_len () < 1)
    return __open_2 (__path, __oflag);

  return __open_alias (__path, __oflag, __va_arg_pack ());
}

The reason GNU C Compiler (eg. gcc) is needed is, at the very least, because of the following code from file /usr/include/sys/cdefs.h:

#if __GNUC_PREREQ (4,3)
# define __warndecl(name, msg) \
  extern void name (void) __attribute__((__warning__ (msg)))
# define __warnattr(msg) __attribute__((__warning__ (msg)))
# define __errordecl(name, msg) \                                                                                               
  extern void name (void) __attribute__((__error__ (msg)))
#else
# define __warndecl(name, msg) extern void name (void)
# define __warnattr(msg)
# define __errordecl(name, msg) extern void name (void)
#endif       

that says that gcc version 4.3 is the minimum required for this to work. (FYI: my current version is gcc (GCC) 8.3.0)

So if you try clang version 8.0.0 (tags/RELEASE_800/final) Target: x86_64-pc-linux-gnu, you don't get the compile error:

$ clang -D_FORTIFY_SOURCE=2 -O1 a.c

(no output here even with -, compilation succeeded: a.out created) because this clang version defines __GNUC__ to be 4 and __GNUC_MINOR__ to be 2 thus 4.2 is just shy of the 4.3 required for it to work; and forcing eg. 8.3 won't work:

$ clang -D_FORTIFY_SOURCE=1 -D__GNUC__=8 -D__GNUC_MINOR__=8 -O1 a.c
In file included from <built-in>:355:
<command line>:2:9: warning: '__GNUC__' macro redefined [-Wmacro-redefined]
#define __GNUC__ 8
        ^
<built-in>:9:9: note: previous definition is here
#define __GNUC__ 4
        ^
In file included from <built-in>:355:
<command line>:3:9: warning: '__GNUC_MINOR__' macro redefined [-Wmacro-redefined]
#define __GNUC_MINOR__ 8
        ^
<built-in>:7:9: note: previous definition is here
#define __GNUC_MINOR__ 2
        ^
2 warnings generated.

The above source codes come from glibc 2.29.9000.r269.g1f50f2ad85-1 package on Arch Linux. ie.

/usr/include/sys/cdefs.h is owned by glibc 2.29.9000.r269.g1f50f2ad85-1
/usr/include/bits/fcntl2.h is owned by glibc 2.29.9000.r269.g1f50f2ad85-1

PS: without _FORTIFY_SOURCE, you can get random modes on each program run, like I did:

$ ./go
-r-x--s--T 1 user user 0 May 17 17:22 /tmp/broken_perms.log
$ ./go
---sr-s--- 1 user user 0 May 17 17:23 /tmp/broken_perms.log
$ ./go
-rws--x--- 1 user user 0 May 17 17:23 /tmp/broken_perms.log
$ ./go
--wsr-x--T 1 user user 0 May 17 17:23 /tmp/broken_perms.log

Answer 6

We can use C macros to catch this problem.

#undef open
#define open(a, b, c) open(a, b, c)

Now you can't call open without three arguments.

This is similar to writing macros for struct initializers, to make sure users don't neglect to initialize some members:

#define foo_initializer(a, b, c) { .x = (a), .y = (b), .z = (c) }

If later we add a new member w, we can extend foo_initializer with a new argument. When we recompile the code base, the compiler will find all the places where it's only being given three arguments. Whereas "naked" initializers that neglect to initialize w will continue to compile cleanly.