22

I need to copy a file from one location to another, and I need to throw an exception (or at least somehow recognise) if the file already exists at the destination (no overwriting).

I can check first with os.path.exists() but it's extremely important that the file cannot be created in the small amount of time between checking and copying.

Is there a built-in way of doing this, or is there a way to define an action as atomic?

Ivy
  • 3,393
  • 11
  • 33
  • 46
  • Is it only creation of the destination that needs to be atomic, but also having the source contents, as read, represent only a single point-in-time? – Charles Duffy Jul 23 '12 at 14:57
  • Just the creation. I'm writing a program which copies a zone file to /tmp, makes the required changes, and then copies it back at the end. I just need to make sure if two people try and edit at the same time, one of them doesn't lose their changes. – Ivy Jul 23 '12 at 15:10
  • 2
    Be aware that `rename()` is only atomic if the source and destination are on the same filesystem -- so you might want to create your temporary file in the destination directory, not in `/tmp`. – Charles Duffy Jul 23 '12 at 15:26

3 Answers3

20

There is in fact a way to do this, atomically and safely, provided all actors do it the same way. It's an adaptation of the lock-free whack-a-mole algorithm, and not entirely trivial, so feel free to go with "no" as the general answer ;)

What to do

  1. Check whether the file already exists. Stop if it does.
  2. Generate a unique ID
  3. Copy the source file to the target folder with a temporary name, say, <target>.<UUID>.tmp.
  4. Rename† the copy <target>-<UUID>.mole.tmp.
  5. Look for any other files matching the pattern <target>-*.mole.tmp.
    • If their UUID compares greater than yours, attempt to delete it. (Don't worry if it's gone.)
    • If their UUID compares less than yours, attempt to delete your own. (Again, don't worry if it's gone.) From now on, treat their UUID as if it were your own.
  6. Check again to see if the destination file already exists. If so, attempt to delete your temporary file. (Don't worry if it's gone. Remember your UUID may have changed in step 5.)
  7. If you didn't already attempt to delete it in step 6, attempt to rename your temporary file to its final name, <target>. (Don't worry if it's gone, just jump back to step 5.)

You're done!

How it works

Imagine each candidate source file is a mole coming out of its hole. Half-way out, it pauses and whacks any competing moles back into the ground, before checking no other mole has fully emerged. If you run this through in your head, you should see that only one mole will ever make it all the way out. To prevent this system from livelocking, we add a total ordering on which mole can whack which. Bam! A  PhD thesis  lock-free algorithm.

† Step 4 may look unnecessary—why not just use that name in the first place? However, another process may "adopt" your  mole  file in step 5, and make it the winner in step 7, so it's very important that you're not still writing out the contents! Renames on the same file system are atomic, so step 4 is safe.

Alice Purcell
  • 12,622
  • 6
  • 51
  • 57
13

There is no way to do this; file copy operations are never atomic and there is no way to make them.

But you can write the file under a random, temporary name and then rename it. Rename operations have to be atomic. If the file already exists, the rename will fail and you'll get an error.

[EDIT2] rename() is only atomic if you do it in the same file system. The safe way is to create the new file in the same folder as the destination.

[EDIT] There is a lot of discussion whether rename is always atomic or not and about the overwrite behavior. So I dug up some resources.

On Linux, if the destination exists and both source and destination are files, then the destination is silently overwritten (man page). So I was wrong there.

But rename(2) still guarantees that either the original file or the new file remain valid if something goes wrong, so the operation is atomic in the sense that it can't corrupt data. It's not atomic in the sense that it prevents two processes from doing the same rename at the same time and you can predict the result. One will win but you can't tell which.

On Windows, if another process is currently writing the file, you get an error if you try to open it for writing, so one advantage for Windows, here.

If your computer crashes while the operation is written to disk, the implementation of the file system will decide how much data gets corrupted. There is nothing an application could do about this. So stop whining already :-)

There is also no other approach that works better or even just as well as this one.

You could use file locking instead. But that would just make everything more complex and yield no additional advantages (besides being more complicated which some people do see as a huge advantage for some reason). And you'd add a lot of nice corner cases when your file is on a network drive.

You could use open(2) with the mode O_CREAT which would make the function fail if the file already exists. But that wouldn't prevent a second process to delete the file and writing their own copy.

Or you could create a lock directory since creating directories has to be atomic as well. But that would not buy you much, either. You'd have to write the locking code yourself and make absolutely, 100% sure that you really, really always delete the lock directory in case of disaster - which you can't.

Aaron Digulla
  • 321,842
  • 108
  • 597
  • 820
  • Only if you flush and sync before renaming. – dcolish Jul 23 '12 at 14:47
  • Out of curiosity, how would the renaming portion of this work? `os.rename` won't work on Unix. – mgilson Jul 23 '12 at 14:49
  • @dcolish Don't need to sync -- flushing or `close()`ing is enough. – Charles Duffy Jul 23 '12 at 14:49
  • 1
    @mgilson Pardon? Certainly it does. – Charles Duffy Jul 23 '12 at 14:49
  • @CharlesDuffy Not true, see note. http://docs.python.org/library/stdtypes.html?highlight=file.flush#file.flush – dcolish Jul 23 '12 at 14:50
  • 2
    @dcolish That's to write to disk. You don't need to be out to disk to be atomic from a filesystem operations perspective, just to be crash-resistant. – Charles Duffy Jul 23 '12 at 14:50
  • @CharlesDuffy -- from the docs: "On Unix, if dst exists and is a file, it will be replaced silently if the user has permission." Do I not understand that properly? – mgilson Jul 23 '12 at 14:51
  • @CharlesDuffy, sure if I didn't care about all my data. – dcolish Jul 23 '12 at 14:51
  • @mgilson That's exactly the behavior we want -- silently, atomically replacing the destination with the file having the new contents. – Charles Duffy Jul 23 '12 at 14:51
  • @dcolish Puh-leeze. If you have both data and metadata journaled, you'll have something there on recovery -- whether it's the old version or the new version, they're both valid. If you don't have data journaling, that was your own decision not to care in setting up the filesystem's mount flags. – Charles Duffy Jul 23 '12 at 14:52
  • Rename is **not** atomic, at least for Btrfs. – Mechanical snail Jul 23 '12 at 14:53
  • @CharlesDuffy You know thats not true. Every database uses an explicit sync when writing data. They do this for a very good reason because it is expensive. – dcolish Jul 23 '12 at 14:53
  • @Mechanicalsnail https://btrfs.wiki.kernel.org/index.php/FAQ#What_are_the_crash_guarantees_of_overwrite-by-rename.3F explicitly documents that it is. – Charles Duffy Jul 23 '12 at 14:53
  • @dcolish They do that because most people don't journal their data, but only the metadata. Yes, data journaling _is_ expensive, but eh, that's your choice. – Charles Duffy Jul 23 '12 at 14:54
  • @CharlesDuffy: That's overwrite-by-rename. It's only atomic if the destination file already exists. – Mechanical snail Jul 23 '12 at 14:55
  • @Mechanicalsnail Fair 'nuff. That's btrfs playing fast-and-loose with POSIX, though. See the "rationale" section of http://pubs.opengroup.org/onlinepubs/009695399/functions/rename.html – Charles Duffy Jul 23 '12 at 14:56
  • @CharlesDuffy True, data journaling will help, but it happens at a given interval or whenever the kernel thinks it is time not necessarily when you make the changes. – dcolish Jul 23 '12 at 14:59
  • Hmm? Yes, the journal isn't continually flushed, but the point is that the ordering of the updates be preserved (metadata update gets applied only if the data one is there first), not that we actually have a journal replay complete before the call returns. Anyhow -- I concede that in the general case (metadata-only journaling), an explicit flush to disk before a rename-based atomic update is the Right Thing when one cares more about durability than performance. – Charles Duffy Jul 23 '12 at 15:23
  • But what about atomicity in the case of power failure? or zombie apocalypse? – msw Jul 23 '12 at 15:29
1

A while back my team needed a mechanism for atomic writes in Python and we came up the following code (also available in a gist):

def copy_with_metadata(source, target):
    """Copy file with all its permissions and metadata.
    
    Lifted from https://stackoverflow.com/a/43761127/2860309
    :param source: source file name
    :param target: target file name
    """
    # copy content, stat-info (mode too), timestamps...
    shutil.copy2(source, target)
    # copy owner and group
    st = os.stat(source)
    os.chown(target, st[stat.ST_UID], st[stat.ST_GID])

def atomic_write(file_contents, target_file_path, mode="w"):
    """Write to a temporary file and rename it to avoid file corruption.
    Attribution: @therightstuff, @deichrenner, @hrudham
    :param file_contents: contents to be written to file
    :param target_file_path: the file to be created or replaced
    :param mode: the file mode defaults to "w", only "w" and "a" are supported
    """
    # Use the same directory as the destination file so that moving it across
    # file systems does not pose a problem.
    temp_file = tempfile.NamedTemporaryFile(
        delete=False,
        dir=os.path.dirname(target_file_path))
    try:
        # preserve file metadata if it already exists
        if os.path.exists(target_file_path):
            copy_with_metadata(target_file_path, temp_file.name)
        with open(temp_file.name, mode) as f:
            f.write(file_contents)
            f.flush()
            os.fsync(f.fileno())

        os.replace(temp_file.name, target_file_path)
    finally:
        if os.path.exists(temp_file.name):
            try:
                os.unlink(temp_file.name)
            except:
                pass

With this code, copying a file atomically is as simple as reading it into a variable and then sending it to atomic_write.

The comments should provide a good idea of what's going on but I also wrote up this more complete explanation on Medium for anyone interested.

therightstuff
  • 833
  • 1
  • 16
  • 21