3

Does the OS handle it correctly?

Or will I have to call flock()?

Stephen P
  • 14,422
  • 2
  • 43
  • 67
user855
  • 19,048
  • 38
  • 98
  • 162

5 Answers5

10

Although the OS won't crash, and the filesystem won't be corrupted, calls to write() are NOT guarenteed to be atomic, unless the file descriptor in question is a pipe, and the amount of data to be written is PIPE_MAX bytes or less. The relevant part of the standard:

An attempt to write to a pipe or FIFO has several major characteristics:

  • Atomic/non-atomic: A write is atomic if the whole amount written in one operation is not interleaved with data from any other process. This is useful when there are multiple writers sending data to a single reader. Applications need to know how large a write request can be expected to be performed atomically. This maximum is called {PIPE_BUF}. This volume of IEEE Std 1003.1-2001 does not say whether write requests for more than {PIPE_BUF} bytes are atomic, but requires that writes of {PIPE_BUF} or fewer bytes shall be atomic.

[...]

As such, in principle, you must lock with simultaneous writers, or your written data may get mixed up and out of order (even within the same write) or you may have multiple writes overwriting each other. However, there is an exception - if you pass O_APPEND, your writes will be effectively atomic:

If the O_APPEND flag of the file status flags is set, the file offset shall be set to the end of the file prior to each write and no intervening file modification operation shall occur between changing the file offset and the write operation.

Although this is not necessarily atomic with respect to non-O_APPEND writes, or simultaneous reads, if all writers use O_APPEND, and you synchronize somehow before doing a read, you should be okay.

Community
  • 1
  • 1
bdonlan
  • 224,562
  • 31
  • 268
  • 324
  • 1
    I don't think `O_APPEND` makes writes atomic. All your citation says is that the write will always happen at the end of the file. Two processes writing simultaneously with `O_APPEND` will not clobber each other's writes, but they could still come out interleaved. – R.. GitHub STOP HELPING ICE Jan 25 '12 at 03:29
  • @R.., this is true, but if two processes each issue a _single_ write of PIPE_BUF bytes or less to the same file opened (on both ends) with O_APPEND, then the writes will not be interleaved – bdonlan Jan 25 '12 at 06:27
  • The O_APPEND text has nothing to do with pipes. Pipes are always append-like. That text is about ordinary files, and the PIPE_BUF text does not apply to ordinary files. – R.. GitHub STOP HELPING ICE Jan 25 '12 at 06:37
  • @R.., I see your point with PIPE_BUF being irrelevant, but in practice, the latter quote means that the write must _complete_ before an intervening write occurs. Interpreting it to say 'no intervening modification operation shall occur between changing the file offset and writing _at least one byte_' is a bit pedantic, and I don't think I've ever seen a system that does it this way. – bdonlan Jan 26 '12 at 06:37
  • 5
    Keep in mind that `write` is interruptable by signals (including the unblockable SIGSTOP) and while this won't give EINTR unless you setup a non-restarting signal handler, it will cause write to return if it has already completed a partial write. Thus there's no way you can ensure atomicity. Even without signals, `write` with `O_APPEND` should be treated as if it's made up of `N` smaller writes, each of which obeys the condition of atomicity between the seek and writing, but which can be non-atomic with respect to one another. – R.. GitHub STOP HELPING ICE Jan 26 '12 at 08:07
2

write (and writev, too) guarantee atomicity.

Which means if two threads or processes write simultaneously, you do not have a guarantee which one writes first. But you do have the guarantee that anything that is in one syscall will not be intermingled with data from the other one.

Insofar it will always work correctly, but not necessarily in the way you expect (if you assume that process A comes before process B).

Damon
  • 67,688
  • 20
  • 135
  • 185
  • My only requirement is that if Thread-A writes a bunch of lines (say lines-A) and thread-B writes a set of lines (say lines-B) then the output is either entire lines-A followed by lines-B or entire lines-B followed by lines-A. I don't want half of lines-A to get written in between the lines-B. As long as lines-A and lines-B get committed fully without the other thread dumping its output in between I don't care. I don't care about the ordering of relative ordering between lines-A and lines-B. (lines-A, lines-B) is acceptable. (lines-B, lines-A) is also acceptable – user855 Aug 29 '11 at 22:23
  • `write` and `writev` guarantee atomicity only with pipes or FIFOs with writes smaller than `PIPE_MAX`, I believe - unless you have a citation otherwise? – bdonlan Aug 30 '11 at 01:46
  • 1
    @bdonlan: There is for example this wording for `writev` (did not search for the one for `write`): _"The data transfers performed by readv() and writev() are atomic: the data written by writev() is written as a single block that is not intermingled with output from writes in other processes (but see pipe(7) for an exception)"_ – Damon Aug 30 '11 at 09:29
  • 1
    @ajay: You can get any possible combination of A and B lines depending on many factors that you can't immediately tell (IO priorities, process priorities, signals and syscall restart, cosmic radiation), but you should never (assuming no kernel/driver bugs) get a half A line with a half B line or some other garbage. – Damon Aug 30 '11 at 09:33
  • 1
    The manpage says that: "Note that is not an error for a successful call to transfer fewer bytes than requested". If process A makes a partial write, its data might be intermingled with data from process B. – ysdx Jan 02 '17 at 15:05
1

Of course the kernel will handle it correctly, for the kernel’s idea of correctness — which is by definition correct.

If you have a set of coöperating flockers, then you can use the kernel to queue everyone up. But remember that flock has nothing to do with I/O: it will not stop someone else from writing the file. It will at most only interfere with other flockers.

tchrist
  • 78,834
  • 30
  • 123
  • 180
1

Yes of course it will work correctly. It won't crash the OS or the process.

Whether it makes any sense, depends on the way the application(s) are written an what the file's purpose is.

If the file is opened by all processes as append-only, each process (notionally) does an atomic seek-to-end before each write; these are guaranteed not to overwrite each others' data (but of course, the order is nondeterministic).

In any case, if you use a library which potentially splits a single logical write into several write syscalls, expect trouble.

MarkR
  • 62,604
  • 14
  • 116
  • 151
0

write(), writev(), read(), readv() can generate partial writes/reads where the amount of data transferred is smaller than what was requested.

Quoting the Linux man page for writev():

Note that is not an error for a successful call to transfer fewer bytes than requested

Quoting the POSIX man page:

If write() is interrupted by a signal after it successfully writes some data, it shall return the number of bytes written.

AFAIU, O_APPEND does not help in this regard because it does not prevent partial writes: it only ensures that whatever data is written is appended at the end of the file.

See this bug report from the Linux kernel:

A process is writing a messages to the file. [...] the writes [...] can be split in two. [...] So if the signal arrives [...] the write is interrupted. [...] this is perfectly valid behavior as far as spec (POSIX, SUS,...) is concerned

FIFOs and PIPE writes smaller than PIPE_MAX however are guaranteed to be atomic.

ysdx
  • 8,889
  • 1
  • 38
  • 51