1

I have an existing file and I'd like to append data to it and make sure it can never (or almost never) get corrupted, even if something fails during writing of the appended data.

One method for ensuring files won't get corrupted it to write the data to a temp file, and then rename/mv the temp file to the original file.

But doing so with append is more tricky.

I have the whole file content in memory (it's not a huge file), so I have two options in mind:

  • Copy the original file to a temp file, append the data to the temp file and then mv/rename the temp file to the original file
  • Write the whole content of the file (including the data I want to append) to a temp file and then mv/rename the temp file to the original file

The downside of both options is that they're slower than just append the data to the original file. Are there better ways to do this?

If not, which option is faster?

I need this to work on Windows, Linux and MacOS.

I'm not sure if the programming language I'm using is relevant, but I'm using Rust to write the data.

seladb
  • 852
  • 1
  • 13
  • 29
  • 2
    I can't see the difference between the two options. – Martin Heralecký Feb 03 '19 at 11:30
  • I'm not an OS / file-system expert, so I thought maybe one of them is faster due to file system optimizations. Are you sure both of them have the same performance? – seladb Feb 03 '19 at 11:32
  • Copying the file by asking kernel to do it (not opening the original, looping over it to read into buffer and then writing) will probably be faster (1. option) than reading the whole file into memory and then writing (2. option). The "append-part" stays the same for both options. – Martin Heralecký Feb 03 '19 at 11:41
  • 2
    Another option is to add checksums for each append which will be checked when reading. This is more suitable for large files. See https://blog.gocept.com/2013/07/15/reliable-file-updates-with-python/. – Martin Heralecký Feb 03 '19 at 11:44
  • See: [Prepend line to beginning of file](https://stackoverflow.com/questions/43441166/prepend-line-to-beginning-of-file) – Peter Hall Feb 03 '19 at 15:08

0 Answers0