2

I'm writing a Windows service to process files created by another process over which I have no control. These files could potentially be very large (hundreds of megabytes).

I need to process and then delete the files after they've been created.

All the files will be written to a particular directory (by just a straight file copy as far as I'm aware), so I can just periodically iterate over the files in that directory, process them and then delete them.

What I'm worried about is what happens if my service queries the directory during the writing of a large file? Will the file show up to my service? Will it be locked so that I can't get read access? Do I need to do anything special to check whether the file has finished copying, or can I just query File.Exists() or try to Open it with FileAccess.Read. How does Windows mark a file that is in the process of being copied?

Jackson Pope
  • 14,520
  • 6
  • 56
  • 80

3 Answers3

2

If this was plain win32 you would try to open the file, with CreateFile(), and a share mode that denied write access to others. That would have to fail if the other program was still writing the file since you can't deny write access when the file is already opened with write access. If it succeeds, you know that the other process has finished.

In .net you could, for example, create a FileStream using one of the constructors that receives a FileShare parameter. This will ultimately map down to the underlying CreateFile() API.

David Heffernan
  • 601,492
  • 42
  • 1,072
  • 1,490
1

AFAIK there is no special mark in a file to indicate that it is being copied, other than it will have a write lock. It is standard practice in this situation to try to open the file yourself with a write lock (e.g., FileShare.Read), and to catch any IOException that occurs because the file is already locked; in which case, pause for a bit (Thread.Sleep) before retrying the file open. You may want to limit the number of retries (to prevent an infinite loop in case the existing file lock is never released).

You say you want to process the files and then delete them? To avoid a race with another process/thread writing to the same file whilst you are processing it/deleting it, you should think of your processing/deleting as an atomic operation, e.g., something like this:

string sourcePath = @"C:\temp1\temp.txt";
string targetPath = @"C:\temp2\temp.txt";
int attempt = 0;
const int maxAttempts = 3;
bool moved = false;
do
{
    try
    {
        File.Move(sourcePath, targetPath);
        moved = true;
    }
    catch (IOException)
    {
        if (attempt < maxAttempts)
        {
            System.Threading.Thread.Sleep(1000);
            attempt++;
        }
    }
} while (!moved && attempt < maxAttempts);

if (moved)
{
     ProcessFile(targetPath);
     File.Delete(targetPath);
}
else
{
    throw new InvalidOperationException("Unable to process '" + sourcePath + "'.");
}

Edit: I see you say the files could be very large, so you shouldn't use File.ReadAllText. You could instead try to move the files to another directory - this will throw an exception is the file is still locked by the other process. You only process the file if you successfully move it. This also has the benefit of removing the file from the input directory.

Polyfun
  • 9,479
  • 4
  • 31
  • 39
0

Write the file with a temporary filename and then rename the file.

The renaming is a atomary process, so your service which process the files should be ok. Just make sure that the service skips the temporary filenames.

Tobias Schittkowski
  • 2,221
  • 2
  • 16
  • 25
  • Is this what Windows does, or what you recommend what I should do? I don't have control over the write process. – Jackson Pope Mar 07 '11 at 10:20
  • "Write the file with a temporary filename and then rename the file" Does windows guarantee that this works? Do you have any official documentation stating that? On Linux the rename can happen before the writing is finished. – CodesInChaos Mar 07 '11 at 11:18
  • I would implement it in this way, but you need access to the write process of course... – Tobias Schittkowski Mar 09 '11 at 13:45
  • Atomicity of the rename process: http://stackoverflow.com/questions/167414/is-an-atomic-file-rename-with-overwrite-possible-on-windows – Tobias Schittkowski Mar 09 '11 at 13:46