4

Based on Is file append atomic in UNIX? and other sources, it looks like on modern Linux, I can open a file in append mode and write small chunks (< PIPE_BUF) to it from multiple processes without worrying about tearing.

Are those limits extended to Ruby with syswrite? Specifically for this code:

f = File.new('...', 'a')
f.syswrite("short string\n")

Can I expect the write to not interleave with other process writing the same way? Or is there some buffering / potential splitting I'm not aware of yet?

Assuming ruby >= 2.3

viraptor
  • 33,322
  • 10
  • 107
  • 191

2 Answers2

2

I recently researched this very topic in order to implement the File appender in Rackstash.

You can find tests for this in the specs which I originally adapted from a blog post on this topic, whose code unfortunately is non-conclusive unfortunately since the author does not write to the file directly but through a pipe. Please read the comments there.

With (1) modern operating systems and (2) their usual local filesystems, the OS guarantees that concurrent appends from multiple processes do write interleave data.

The main points here are:

  • You need a reasonably modern operating system. Very old (or exotic) systems had lesser guarantees. Here, you might have to lock the file explicitly with e.g. File#flock.
  • You need to use a compatible filesystem. Most local filesystems like HFS, APFS, NTFS, and the usual Linux filesystems like ext3, ext4, btrfs, ... should be safe. SMB is one of the few network filesystems that also guarantees this. NFS and most FUSE filesystems are not safe in this regard.

Note that this mechanism does not guarantee that concurrent readers always read full writes. While the writes themselves are never interleaved, readers might read the partial results of in-progress writes.

As far as my understanding (and my tests) goes, the atomically writable size isn't even restricted to PIPE_SIZE. This limit only applies when writing to a pipe like a socket or e.g. STDOUT instead of a real file.

Unfortunately, authoritative information on this topic is rather sparse. Most articles (and SO answers) on this topic conflate strict appending with random writes. When not strictly appending (by opening the file in append-only mode), the guarantees are null and void.

Thus, to answer your specific question: Yes, your code from your question should be safe when writing to a local filesystem on modern operating systems. I think syswrite bypasses the file buffer already. To be sure, you should also set f.sync = true before writing to it to completely disable any buffering.

Note that you should still use a Mutex (or similar) if you plan to write to the single opened file from multiple threads in your process (since the append-guarantees of the OS are only valid for concurrent writes to different file-descriptors; it can not discern overlapping writes by the same process to the same file descriptor).

Holger Just
  • 52,918
  • 14
  • 115
  • 123
1

I would not assume that. syswrite calls the write POSIX function, which make no claims about atomicity when dealing with files.

See: Are POSIX' read() and write() system calls atomic?

And Understanding concurrent file writes from multiple processes

Tl;dr- you should implement some concurrency control in your app to synchronize this access.

Joe
  • 41,484
  • 20
  • 104
  • 125
  • That's the reason I mentioned modern linux. While POSIX [doesn't guarantee](http://pubs.opengroup.org/onlinepubs/9699919799/functions/open.html) anything, Linux is [more strict](http://man7.org/linux/man-pages/man2/open.2.html): "The modification of the file offset and the write operation are performed as a single atomic step." – viraptor Mar 07 '18 at 03:18
  • @viraptor YARV literally calls plain old [write](https://github.com/ruby/ruby/blob/96db72ce38b27799dd8e80ca00696e41234db6ba/io.c#L951). If you want to be super sure of this behavior, you can compile Ruby yourself on the system you need this behavior on. – Max Mar 07 '18 at 16:44