1

I have a database file which is about 1GB and it is synced to disk. I want to update a little piece of 20 bytes in the middle of the file. I want to do this using classical syscalls (open/lseek/write/close). The operation must be atomic in case of a power cut. A journal file can be used.

I see the problem when calling write(fd, buf, 20) to update the db file. I think there is no guarantee that other parts than the 20 bytes will not be damaged in case of a power cut. For example the beginning of the file may be damaged. There's simply no guarantee.

MySQL or SQLite claim to be able to perform such operations.

zomega
  • 1,538
  • 8
  • 26
  • See answer https://stackoverflow.com/a/53340087/18980756 but look at the comments there in order to include using `fsync` in your solution. – Franck May 15 '22 at 13:05
  • I already saw this thread. It's about writing a file from scratch and not about updating parts of a file. – zomega May 15 '22 at 13:08
  • One classical problem is that database storage files already try to be a filesystem in themselves. You can do it like that, that would indeed mean you would do journaling yourself, internally within the file format, or externally using separate files. Or you can just trust the journaling of the filesystem the files are on. – Cheatah May 15 '22 at 13:09
  • @somega You have no way to avoid the copy for what you describe. See also http://www.microhowto.info/howto/atomically_rewrite_the_content_of_a_file.html – Franck May 15 '22 at 13:10
  • @somega The microhowto just describes in detail what the linked answer above suggests. – Franck May 15 '22 at 13:17
  • @Franck Well, you will have to avoid making copies if you want to have any kind of performance. That is why most database format use fixed size records or blocks, and basically do lots of internal bookkeeping. This is not the type of operation you would want to do by copying the file. – Cheatah May 15 '22 at 13:18
  • @Cheatah I agree with you that for performance reasons and for your own databases implementation you do not want to do this. This means that you do your bookkeeping in another file where you apply this mechanism. And you can *not* update your few bytes in the large file directly atomically. Using your bookkeeping file you can appear to do so. – Franck May 15 '22 at 13:24
  • @somega An explanation can be found at https://en.wikipedia.org/wiki/Atomicity_(database_systems)#Implementation – Franck May 15 '22 at 13:29

0 Answers0