Reliable file saving (File.Replace) in a busy environment

Question

I am working on server software that periodically needs to save data to disk. I need to make sure that the old file is overwritten, and that the file cannot get corrupted (e.g. only partially overwritten) in case of unexpected circumstances.

I've adopted the following pattern:

string tempFileName = Path.GetTempFileName();
// ...write out the data to temporary file...
MoveOrReplaceFile(tempFileName, fileName);

...where MoveOrReplaceFile is:

public static void MoveOrReplaceFile( string source, string destination ) {
    if (source == null) throw new ArgumentNullException("source");
    if (destination == null) throw new ArgumentNullException("destination");
    if (File.Exists(destination)) {
        // File.Replace does not work across volumes
        if (Path.GetPathRoot(Path.GetFullPath(source)) == Path.GetPathRoot(Path.GetFullPath(destination))) {
            File.Replace(source, destination, null, true);
        } else {
            File.Copy(source, destination, true);
        }
    } else {
        File.Move(source, destination);
    }
}

This works well as long as the server has exclusive access to files. However, File.Replace appears to be very sensitive to external access to files. Any time my software runs on a system with an antivirus or a real-time backup system, random File.Replace errors start popping up:

System.IO.IOException: Unable to remove the file to be replaced.

Here are some possible causes that I've eliminated:

Unreleased file handles: using() ensures that all file handles are released as soon as possible.
Threading issues: lock() guards all access to each file.
Different disk volumes: File.Replace() fails when used across disk volumes. My method checks this already, and falls back to File.Copy().

And here are some suggestions that I've come across, and why I'd rather not use them:

Volume Shadow Copy Service: This only works as long as the problematic third-party software (backup and antivirus monitors, etc) also use VSS. Using VSS requires tons of P/Invoke, and has platform-specific issues.
Locking files: In C#, locking a file requires maintaining a FileStream open. It would keep third-party software out, but 1) I still won't be able to replace the file using File.Replace, and 2) Like I mentioned above, I'd rather write to a temporary file first, to avoid accidental corruption.

I'd appreciate any input on either getting File.Replace to work every time or, more generally, saving/overwriting files on disk reliably.

Do you expect `MoveOrReplaceFile` to be run concurrently (meaning accessed by multiple threads of *your* application or even from multiple instances of your application)? — M.Babcock, Jan 22 '12 at 02:19
My own code never calls MoveOrReplaceFile concurrently, but there are other processes outside my control that may attempt to read the file. Those random read accesses are what causes File.Replace to fail. — matvei, Jan 23 '12 at 11:07

Hans Passant · Accepted Answer · 2012-01-22T21:09:41.150

30

You really want to use the 3rd parameter, the backup file name. That allows Windows to simply rename the original file without having to delete it. Deleting will fail if any other process has the file opened without delete sharing, renaming is never a problem. You could then delete it yourself after the Replace() call and ignore an error. Also delete it before the Replace() call so the rename won't fail and you'll cleanup failed earlier attempts. So roughly:

string backup = destination + ".bak";
File.Delete(backup);
File.Replace(source, destination, backup, true);
try {
    File.Delete(backup);
}
catch {
    // optional:
    filesToDeleteLater.Add(backup);
}

edited Jan 22 '12 at 21:09

answered Jan 22 '12 at 20:06

Hans Passant

922,412
146
1,693
2,536

Hans, could you show the code for this pattern? This might then become the canonical answer for this question in the future. – John Saunders Jan 22 '12 at 20:10
This is precisely what I was looking for! Thank very much. – matvei Jan 23 '12 at 11:09
2

Doesn't this just shift the problem down one level? What happens if the backup file can't be deleted? – grantnz Sep 17 '12 at 03:20
2

Nothing beyond having an extra .bak file survive that will be deleted the next time. Key point is that the primary operation won't fail. – Hans Passant Sep 17 '12 at 03:59
2

A warning for someone who simply copy this code. Accordingly to the documentation of "File.Delete(..)" it will fail if the backup file does not exist, so a check is needed. – Robert Feb 27 '15 at 08:47
3

No. Quote from the MSDN article: "If the file to be deleted does not exist, no exception is thrown." – Hans Passant Feb 27 '15 at 08:49
[It seems that](http://stackoverflow.com/questions/43574311/renaming-a-file-open-in-another-process-without-file-share-delete?noredirect=1&lq=1#comment74199621_43574369) `File.Replace` is implemented using [`ReplaceFile`](https://msdn.microsoft.com/en-us/library/windows/desktop/aa365512(v=vs.85).aspx). When I used ReplaceFile, I was still [unable to rename a file open in another process](http://stackoverflow.com/questions/43578567/why-does-replacefile-fail-with-error-sharing-violation). Is there anything else I need to do to make it work? – max Apr 24 '17 at 02:33

score 2 · Answer 2 · answered Jan 22 '12 at 19:43

There are several possible approaches, here some of them:

Use a "lock" file - a temporary file that is created before the operation and indicates other writers (or readers) that the file is being modified and thus exclusively locked. After the operation complete - remove the lock file. This method assumes that the file-creation command is atomic.
Use NTFS transactional API (if appropriate).
Create a link to the file, write the changed file under a random name (for example Guid.NewGuid()) - and then remap the link to the new file. All readers will access the file through the link (which name is known).

Of course all 3 approaches have their own drawbacks and advantages

score 1 · Answer 3 · answered Jan 22 '12 at 02:51

1

If the software is writing to an NTFS partition then try using Transactional NTFS. You can use AlphFS for a .NET wrapper to the API. That is probably the most reliable way to write files and prevent corruption.

answered Jan 22 '12 at 02:51

Scott Lerch

2,620
1
23
34

Works only on Vista and later, though. – CodesInChaos Jan 22 '12 at 21:30
1

`File.Replace()` under mono magically does stuff like using POSIX `rename()` on other OSes. Wouldn’t moving away from built-in classes complicate portability? Also, if `File.Replace()` actually worked as documented and reliably, it already provides the necessary functionality… – binki Jun 21 '17 at 13:23
any full source code sample using ***AlphaFS*** ? The project move to Github: https://github.com/alphaleonis/AlphaFS – Kiquenet Oct 07 '20 at 08:03
1

Given that Microsoft has deprecated Transactional NTFS (TxF): https://archive.codeplex.com/?p=transactionalfilemgr and https://www.chinhdo.com/20080825/transactional-file-manager/ and too https://github.com/goldfix/Transactional-NTFS-TxF-.NET – Kiquenet Oct 07 '20 at 08:07

Reliable file saving (File.Replace) in a busy environment

3 Answers3

Linked