9

This article states, that

fd = os.open('foo.lock', os.O_CREAT|os.O_EXCL|os.O_RDWR)

"is atomic on most filesystems". Is that true (on Unix and Windows)? On which filesystems?

The docs state that mentioned flags are available on Unix and Windows, so it looks like a tempting, cross-platform method for file locking (the flags O_CREAT and O_EXCL ensure that the calling process creates the file).

Richard Pump
  • 588
  • 3
  • 8
  • 3
    This will fail at random on NFS filesystems even though it isn't supposed to. I would suspect that there are others (e.g. gmail-fs) which won't guarantee exclusivity and it would be extraordinarily difficult to determine if `foo.lock` is on a semantically proper mount. – msw Feb 25 '13 at 16:59
  • 1
    @msw: Good catch. NFS servers, by protocol, don't maintain an "open" state for a file hence `O_EXCL` semantics (whether for open or create) on NFS are subject to races. See http://lwn.net/Articles/252012/ for the technical background. – FrankH. Feb 25 '13 at 17:09
  • 1
    Actually, the [Linux open(2) man-page](http://man7.org/linux/man-pages/man2/open.2.html) says "On NFS, O_EXCL is only supported when using NFSv3 or later on kernel 2.6 or later", but the author of the post pointed by FrankH. still advises not to. – Richard Pump Feb 25 '13 at 20:06

1 Answers1

6

For UN*X-compliant (certified POSIX / IEEE 1003.1 as per the OpenGroup) systems, the behaviour is guaranteed as the OpenGroups specs for open(2) mandate this. Quote:

O_EXCL
If O_CREAT and O_EXCL are set, open() shall fail if the file exists. The check for the existence of the file and the creation of the file if it does not exist shall be atomic with respect to other threads executing open() naming the same filename in the same directory with O_EXCL and O_CREAT set. If O_EXCL and O_CREAT are set, and path names a symbolic link, open() shall fail and set errno to [EEXIST], regardless of the contents of the symbolic link. If O_EXCL is set and O_CREAT is not set, the result is undefined.

The "common" UN*X and UN*X-like systems (Linux, MacOSX, *BSD, Solaris, AIX, HP/UX) surely behave like that.

Since the Windows API doesn't have open() as such, the library function there is necessarily reimplemented in terms of the native API but it's possible to maintain the semantics.

I don't know which widely-used systems wouldn't comply; QNX, while not POSIX-certified, has the same statement in its docs for open(). The *BSD manpages do not explicitly mention the "atomicity" but Free/Net/OpenBSD implement it. Even exotics like SymbianOS (which like Windows doesn't have a UN*X-ish open system call) can do the atomic open/create.

For more interesting results, try to find an operating system / C runtime library which has open() but doesn't implement the above semantics for it... and on which Python would run with threads (got you there, MSDOS ...).

Edit: My post particularly focuses on "which operating systems have this characteristic for open ?" - for which the answer is, "pretty much all of them". Wrt. to filesystems though, the picture is different because network filesystems - whether NFS, SMB/CIFS or others, do not always maintain O_EXCL as this could result in denial-of-service (if a client does an open(..., O_EXCL, ...) and then simply stops talking with the fileserver / is shut down, everyone else would be locked out).

FrankH.
  • 17,675
  • 3
  • 44
  • 63
  • 6
    `O_EXCL` doesn't do that. The only thing `O_EXCL` does is cause the call to fail if the file exists and `O_CREAT` is specified. – Dietrich Epp Feb 25 '13 at 17:32