8

If the file already exists, I want to overwrite it. If it doesn't exist, I want to create it and write to it. I'd prefer to not have to use a 3rd party library like lockfile (which seems to handle all types of locking.)

My initial idea was to:

  1. Write to a temporary file with a randomly generated large id to avoid conflict.
  2. Rename the temp filename -> new path name.
Jonathan Hall
  • 75,165
  • 16
  • 143
  • 189
lf215
  • 1,185
  • 7
  • 41
  • 83
  • 3
    If you restrict yourself to POSIX systems, the Rename is the correct way. Windows however doesn't support this on all versions, and not supported by Go on windows. – JimB May 21 '15 at 22:18
  • @JimB, what do you think about step 1? are there other (significant) steps missing? – lf215 May 21 '15 at 22:45
  • 6
    For #1 I'd use [`ioutil.TempFile`](https://golang.org/pkg/io/ioutil/#TempFile) (with `dir` set to the final directory so that rename won't involve a cross device copy) for creating the temporary rather than trying to generate a random one and handle race conditions yourself. – Dave C May 21 '15 at 22:47
  • 1
    but its not about golang. – Jiang YD May 22 '15 at 00:57
  • For windows, it may be possible to use `ReplaceFile` https://msdn.microsoft.com/en-us/library/windows/desktop/aa365512(v=vs.85).aspx which supports XP to current versions, but you would need to add it alongside shims to the POSIX Rename functions within your own package to make it os-independent. More info at https://msdn.microsoft.com/en-us/library/windows/desktop/hh802690(v=vs.85).aspx – Intermernet May 22 '15 at 09:18

1 Answers1

12

os.Rename calls syscall.Rename which for Linux/UNIXs uses the rename syscall (which is atomic*). On Windows syscall.Rename calls MoveFileW which assuming the source and destination are on the same device (which can be arranged) and the filesystem is NTFS (which is often the case) is atomic*.

I would take care to make sure the source and destination are on the same device so the Linux rename does not fail, and the Windows rename is actually atomic. As Dave C mentions above creating your temporary file (usually using ioutil.TempFile) in the same directory as existing file is the way to go; this is how I do my atomic renames.

This works for me in my use case which is:

  1. One Go process gets updates and renames files to swap updates in.
  2. Another Go process is watching for file updates with fsnotify and re-mmaps the file when it is updated.

In the above use case simply using os.Rename has worked perfectly well for me.

Some further reading:

  1. Is rename() atomic? "Yes and no. rename() is atomic assuming the OS does not crash...."
  2. Is an atomic file rename (with overwrite) possible on Windows?

*Note: I do want to point out that when people talk about atomic filesystem file operations, from an application perspective, they usually mean the operation happens or does not happen (which journaling can help with) from the users perspective. If you are using atomic in the sense of an atomic memory operation, very few filesystem operations (outside of direct I/O [O_DIRECT] one block writes and reads with disk buffering disabled) can be considered truly atomic.

Community
  • 1
  • 1
voidlogic
  • 6,398
  • 2
  • 23
  • 21