How to wait until a file is successfully copied from a network drive before reading from it?

Question

I am currently facing a problem with an application which consists of multiple components. One component of the application periodically checks for new files on a network drive and copies them into a local folder. Another component of the application uses a FileSystemWatcher to watch for any new files in the local folder. If a new file is copied, the Created event of the FileSystemWatcher gets called and the application will then read the file contents and import the file into a database. To prevent the application from trying to read the file before it is fully copied into the local folder, it calls the following function periodically until it returns false:

private bool isFileLocked(string filePath)
{
    try
    {
        if (!File.Exists(filePath))
        {
            return false;
        }

        using (FileStream fs = File.OpenRead(filePath))
        {
        }

        return false;
    }
    catch (IOException)
    {
        return true;
    }
}

Unfortunately this does not seem to work in all cases. Sometimes, I noticed that the file is being read before it is completely written into the local folder. When this happens, the component which tries to copy the file gets the following error:

System.IO.IOException: The process cannot access the file '...' because it is being used by another process.

The component which copies the file is written in PowerShell and uses the following Cmdlet for copying:

Copy-Item $currentfile.FullName -Destination "$destfolder" –Force -ErrorAction Stop

The component which uses the FileSystemWatcher and imports the file is a C# based windows service. How can I prevent it from reading the file before it is fully copied into the local folder?

How about making the methods that you need to wait for "asynchronous"? Mark them with `async` and `await` the operations one by one. If it doesn't make a difference if the thread is blocked, make them Tasks and call `Task.Wait` to ensure it has finished. — thesystem, Oct 19 '21 at 10:49
Does it make any difference if you try to open the file for writing instead? — Matthew Watson, Oct 19 '21 at 10:57
@MatthewWatson I don't know. The problem happens quite rarely (sometimes not at all for a day or two) so I cannot easily test anything in a short period of time. — Chris, Oct 19 '21 at 11:29
@thesystem How would that help when both operations are done by two different applications? I would have to completely rewrite the script which copies the files in C# so I can control both operatios from one place but this isn't a feasible solution. — Chris, Oct 19 '21 at 11:31
Ok, I didn't understand that when first reading your question. Maybe you can find inspiration in here: https://stackoverflow.com/questions/69629410/how-to-wait-until-a-file-is-successfully-copied-from-a-network-drive-before-read?noredirect=1#comment123074808_69629410 — thesystem, Oct 19 '21 at 11:39
For greater reliability I would try to implement some kind of "transactional" scheme. A simple way to do this is to copy the file with a temporary target name like `$currentFile.FullName + '.tmp'`. After the file has been copied, rename it to the final name. When the component using the `FileSystemWatcher` only watches for the final name (ignore "*.tmp"), it can be sure that the file has been copied completely. — zett42, Oct 19 '21 at 11:49
@marsze I don't know because I did not develop this application, I only maintain it. I assume this was done because of problems when using a `FileSystemWatcher` for watching a network folder or something similar. — Chris, Oct 19 '21 at 14:35
@marsze Yeah but I am already waiting until the file is no longer locked in my code, so why does it read from the file anyway before the copying is finished? Is it some sort of timing problem? Is there another event to use instead of create which would be more suitable in this case? — Chris, Oct 20 '21 at 07:39
@marsze The waiting is done by calling `isFileLocked` until it returns false. Only if this is the case, the code will read the file contents. I think what might be happening here is that there is probably a very short period of time between creating and opening the file when copying it, so if isFileLocked is called right between creating and opening it will "steal" the file access and block the copying from writing the file contents. Does that make sense? — Chris, Oct 20 '21 at 08:51

score 0 · Answer 1 · answered Oct 19 '21 at 11:42

0

If you don't worry about little delay - it may solve your trouble:

static void Main(string[] args)
{
    FileSystemWatcher fsw = new FileSystemWatcher("SomePathToFolder");
    fsw.EnableRaisingEvents = true;
    fsw.Created += async (s, a) =>
    {
        while (FileIsLocked(a.FullPath))
        {
            Console.WriteLine($"File {a.Name} is locked!");
            await Task.Delay(TimeSpan.FromSeconds(5)); // 5 seconds delay between checks
        }
  
        Console.WriteLine($"File {a.Name} available!");
  
        // You can put here another delay to be 102% sure that file is free,
        // but I suppose this is too much.
        using (FileStream fs = File.OpenRead(a.FullPath))
        {
            Console.WriteLine($"File {a.Name} opened for reading.");
            // Do what you need
            await Task.Run(() => ImportFileToDatabase(fs));
        }

        Console.WriteLine($"File {a.Name} closed.");
    };


    Console.ReadKey();
}

static bool FileIsLocked(string filePath)
{
    if (!File.Exists(filePath))
        return false;

    try
    {
        using (FileStream fs = File.OpenRead(filePath)) { }
        return false;
    }
    catch { }

    return true;
}

answered Oct 19 '21 at 11:42

Auditive

1,607
1
7
13

Am I correct that this is essentially the same thing I am already doing, but with more delay between checking if the file is accessible and one additional check before finally attempting to read from the file, or am I missing something? – Chris Oct 19 '21 at 11:53
2

Yep, the culprit is just in delay. If it's not critical to have some delay before file processing after copy (insdead of instant (immediate) file processing) - you can just play with delays. Also note @marsze's answer, it combines file opening and waiting for that opening in one async method. – Auditive Oct 19 '21 at 11:55
Yeah I was already thinking about something like this in case there is no better solution but it still feels to me like more of a workaround instead of an actual solution. While this would probably eliminate 99% of all cases where the file is read before it is fully copied, I still dont know if it truly fixes the problem. Shouldnt the existing check I implemented already prevent this problem from happening? – Chris Oct 19 '21 at 11:57
1

To be 102% sure - you should remove PowerShell copying and put it into C#. With streams you would be able to monitor and would have full control over copying. – Auditive Oct 19 '21 at 12:00

marsze · Answer 2 · 2021-10-19T11:53:56.840

Some solutions are suggested here. I've had a similar problem using FileSystemWatcher. This is what I use (simplified):

async Task<FileStream> OpenWaitAsync(string path, TimeSpan interval, CancellationToken cancellationToken = default)
{
    const int ERROR_SHARING_VIOLATION = unchecked((int)0x80070020);
    while (true)
    {
        try
        {
            return File.OpenRead(path);
        }
        catch (IOException ioe) when (ioe.HResult == ERROR_SHARING_VIOLATION)
        {
            await Task.Delay(interval, cancellationToken);
        }
    }
}

How to wait until a file is successfully copied from a network drive before reading from it?

2 Answers2