Best practice for writing big files

Question

I need to write a big file in my project.

What I learned:

I should NOT write the big file directly to the destination path, because this may leave a incomplete file in case the app crash while writing it.
Instead, I should write to a temporary file and move (rename) it. (called atomic file operation)

My code snippet:

[NotNull]
public static async Task WriteAllTextAsync([NotNull] string path, [NotNull] string content) 
{
    string temporaryFilePath = null;
    try {
        temporaryFilePath = Path.GetTempFileName();
        using (var stream = new StreamWriter(temporaryFilePath, true)) {
            await stream.WriteAsync(content).ConfigureAwait(false);
        }            

        File.Delete(path);
        File.Move(temporaryFilePath, path);
    }
    finally {
        if (temporaryFilePath != null) File.Delete(temporaryFilePath);
    }
}

My Question:

The file will be missing if the app crashes between File.Delete and File.Move. Can I avoid this?
Is there any other best practice for writing big files?
Is there any suggestion on my code?

One thing to be aware of when adopting this approach: If the TEMP folder is on a different drive from the final destination folder, then the `File.Move()` will actually have to copy the file, and it it's a very large file that could significantly affect performance. — Matthew Watson, Dec 07 '18 at 08:34
@Matthew Yeah, I'm aware of it. In this case, I have enough reason to conclude the destination and temporary folder are on the same drive. So I only added `finally` block for confirmation (`File.Move` copies when the source and the destination be on the different drive). But it's better if I can avoid performance problems. Is there any solution for this? I'm interested in your idea. — terborac, Dec 07 '18 at 13:40
@SirRufo thank you for yorur informative comment. I didn't know `FileOptions.DeleteOnClose` option. I'm afraid I don't understand how to apply this to my case. You can't `Move` a file before closing (thus before being deleted) it? I'd be appliciated if you help me some more... — terborac, Dec 07 '18 at 13:46
Well, I would write to temp a file, **copy** it to destination (as Move will do when move to a different drive) and drop the temp file. — Sir Rufo, Dec 07 '18 at 13:53
@SirRufo Oh, I see. It seems a good way if the file is not so big. Thank you! — terborac, Dec 07 '18 at 14:00

Jon Skeet · Accepted Answer · 2018-12-07T08:45:33.760

7

The file will be missing if the app crashes between File.Delete and File.Move. Can I avoid this?

Not that I'm aware of, but you can detect it - and if you use a more predictable filename, you can recover from that. It helps if you tweak the process somewhat to use three file names: the target, a "new" file and an "old" file. The process becomes:

Write to "new" file (e.g. foo.txt.new)
Rename the target file to the "old" file (e.g. foo.txt.old)
Rename the "new" file to the target file
Delete the "old" file

You then have three files, each of which may be present or absent. That can help you to detect the situation when you come to read the new file:

No files: Nothing's written data yet
Just target: All is well
Target and new: App crashed while writing new file
Target and old: App failed to delete old file
New and old: App failed after the first rename, but before the second
All three, or just old, or just new: Something very odd is going on! User may have interfered

Note: I was unaware of File.Replace before, but I suspect it's effectively just a simpler and possibly more efficient way of doing the code you're already doing. (That's great - use it!) The recovery process would still be the same though.

edited Dec 07 '18 at 08:45

answered Dec 07 '18 at 08:06

Jon Skeet

1,421,763
867
9,128
9,194

1

It's possible to avoid problems using Transactional NTFS. Microsoft never any NTFS features in .NET so one has to use libraries AlphaFS for this. Using AlphaFS one can write directly to the target file with changes appearing only when the transaction commits – Panagiotis Kanavos Dec 07 '18 at 08:16
1

@PanagiotisKanavos: That sounds like it would make a good answer then :) – Jon Skeet Dec 07 '18 at 08:18
Unfortunately, Microsoft said the feature is deprecated and will be removed in the future because it's not needed with ReFS. That was before they removed ReFS from Windows 10. [The current article](https://learn.microsoft.com/en-us/windows/desktop/fileio/deprecation-of-txf) uses milder wording but still proposes `ReplaceFile` as a "kind-of" alternative – Panagiotis Kanavos Dec 07 '18 at 08:23
@PanagiotisKanavos: Humbug. Thanks for the detail though - always nice to learn more :) – Jon Skeet Dec 07 '18 at 08:23
Side note: one would likely not Delete(current);Rename(new,old) but rather Delete(backup);Rename(current, backup); Rename(new,current);... So you never actually lose the data forever between Delete and Move. – Alexei Levenkov Dec 07 '18 at 08:35
@AlexeiLevenkov: Whoops, yes, that's the form I've done before. Will rewrite the answer on those lines. (Although if you get as far as the Delete you've presumably finished writing the "new" data anyway, so you'd still have that.) – Jon Skeet Dec 07 '18 at 08:38
@JonSkeet thank you for your hands-on answer! I appricitae it! – terborac Dec 07 '18 at 13:58

score 6 · Answer 2 · answered Dec 07 '18 at 08:07

6

You can use File.Replace instead of deleting and moving files. In case of hard fault (electricity cut or something like this) you will always lost data, you have to count with that.

answered Dec 07 '18 at 08:07

110mat110

564
6
26

2

For reference, the CLR implementation of `File.Replace()` uses the [WinAPI method `ReplaceFile()`](https://learn.microsoft.com/en-us/windows/desktop/api/winbase/nf-winbase-replacefilea) – Matthew Watson Dec 07 '18 at 08:43

Best practice for writing big files

2 Answers2