21

Is there a way to dump a buffer to file atomically?

By "atomically" I mean: if for example someone terminates my application during writing, I'd like to have file in either before- or after-writing state, but not in a corrupted intermediate state.

If the answer is "no", then probably it could be done with a really small buffers? For example, can I dump 2 consequent int32_t variables with a single 8 bytes fwrite (on x64 platform), and be sure that both of those int32s are dumped, or neither of them, but not only just one of them?

mambo_sun
  • 509
  • 4
  • 13
  • I think you need to look into unix "Signals". Trapping signals will allow you yo know when your app is being closed. The only thing you can't detected is a `kill -9` which takes down your program without issuing a signal. – Philip Couling Mar 25 '15 at 16:51
  • 17
    A usual workaround is to work on a copy of the file, and swap them atomically at the end. – Marc Glisse Mar 25 '15 at 16:51
  • 4
    Agree with @MarcGlisse the only guaranteed way to prevent corruption is to write a new file and move the new file to the old one as the rename is atomic. – Philip Couling Mar 25 '15 at 16:54
  • 5
    I was about to suggest the same as @MarcGlisse. Just be aware that moving (aka renaming) a file is only an atomic operation if the source and destination are on the same file system. – 5gon12eder Mar 25 '15 at 16:54
  • 3
    @5gon12eder you can't rename a file in C or C++ if the files are not on the same file system. http://www.gnu.org/software/libc/manual/html_node/Renaming-Files.html – Philip Couling Mar 25 '15 at 16:56
  • Thanks a lot for your answers! I'll consider the renaming suggestion. Trapping signals looks tricky in my case, as I am working simultaneously with quite a few files, and will need to add some additional synchronization. Same procedure for every file which keeps them in a good states, suits me better. BTW, great to know that renaming is atomic! – mambo_sun Mar 25 '15 at 17:23
  • There are some guarantees of atomicity, but there are several layers to consider. A `write` system call will be atomic if the number of bytes written is less than PIPE_BUF. But that only gets your data into kernel buffers. It's not on disk yet. Are you asking about getting data from user space , or actually sync'd on a physical device? – William Pursell Nov 16 '18 at 15:03

1 Answers1

23

I recommend writing to a temporary file and then doing a rename(2) on it.

ofstream o("file.tmp"); //Write to a temporary file
o << "my data";
o.close();

//Perform an atomic move operation... needed so readers can't open a partially written file
rename("file.tmp", "file.real");
Gillespie
  • 5,780
  • 3
  • 32
  • 54
  • 5
    This is the right way to go. On POSIX systems, once a file descriptor is opened, its inode is guaranteed to exist until the file descriptor is destroyed. Therefore, readers of the old file are guaranteed to see the same data regardless of writes. The rename operation is guaranteed to be atomic on the same filesystem, therefore, this idiom makes both read and writes concurrent safe. – Jazzwave06 Nov 16 '18 at 14:49
  • Is there a way to do this with multiple files in different directories? Like say I have a/foo.txt and b/c/bar.txt and I make foo.tmp and bar.tmp; how do I make sure that both the real files get updated atomically? – michaelsnowden Jan 31 '20 at 05:31
  • 5
    This is not enough. You must `fsync` the file before you rename it. Otherwise you might end up with an empty file in case of a powercut. – Emil Feb 28 '20 at 00:16
  • 1
    @Emil depends on your requirements. If you need fault tolerance and the files need to be stored on disk, then yes, you need fsync. But if you are just storing the files in `tmpfs` there is no need. – Gillespie Feb 28 '20 at 14:53
  • This won't work if the user doesn't have write permission for the directory the file is in. Even if they can write to the file they may not be able to create a new one which is what rename/mv does. So we don't have a full answer here. – CR. Jun 24 '21 at 22:11
  • You also need to `fsync` the folder after you rename the file. – Vad Aug 15 '23 at 20:28