0

We have a long-running legacy Windows application which cannot be updated for various reasons. This program writes out data files at unpredictable intervals. Before it writes out a new data file, it first renames the previous data file.

Other applications (well, at least one!) need real-time access to the latest data file. The problem is that if the reader program has the file open when the writer program tries to rename it, the writer program crashes. This must not happen!

The obvious solution, fixing the writer program, is not practical. What I would like to find is some Windows service or alternative file system which would transparently copy files as they are written. This way the reader program could access the copied file without disturbing the writer program (it matters much less if the reader program gets a fault).

As far as I can tell, disk mirroring would not solve the file-locking issue we have.

My question: does anyone know of such a service we could use?

EDIT: example code illustrating the problem, even with supposedly non-locking file reads:

const string SrcFilePath = @"C:\Foo.txt";
const string DestFilePath = @"C:\Bar.txt";

void Main()
{
    Task.Run(() =>
    {
        while (true)
        {
            try
            {
                Console.WriteLine("Writer: moving {0} to {1}", SrcFilePath, DestFilePath);
                File.Move(SrcFilePath, DestFilePath);
                Console.WriteLine("Writer: writing {0}", SrcFilePath);
                File.WriteAllText(SrcFilePath, "Hello, World!\n");
            }
            catch (Exception e)
            {
                Console.WriteLine("Writer: error: {0}", e.Message);
            }
            Console.WriteLine("Writer: sleeping.");
            Task.Delay(1000).Wait();
        }
    });
    Task.Run(() =>
    {
        while (true)
            try
            {
                Console.WriteLine("Reader: opening {0}", SrcFilePath);
                var fs = new FileStream(SrcFilePath, FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
                Console.WriteLine("Reader: sleeping...");
                Task.Delay(1500).Wait(); // Simulate a context switch in the middle of a file read.
                Console.WriteLine("Reader: waking up.");
                Console.WriteLine("Reader: closing {0}", SrcFilePath);
                fs.Dispose();
            }
            catch (Exception e)
            {
                Console.WriteLine("Reader: error: {0}", e.Message);
            }
    });
}

This fails like so:

Reader: opening C:\Foo.txt
Reader: sleeping...
Writer: moving C:\Foo.txt to C:\Bar.txt
Writer: error: The process cannot access the file because it is being used by another process.
Writer: sleeping.
Reader: waking up.
Reader: closing C:\Foo.txt
Rafe
  • 5,237
  • 3
  • 23
  • 26
  • Hmm, no, that doesn't make sense. A program can only lock the file data, it cannot lock the directory entry. My crystal ball says that the real reason it bombs is because it tries to append data to the existing file. You must open the file with FILE_SHARE_READ | FILE_SHARE_WRITE to prevent that from failing. Forgetting to share write access being the troublemaker, everybody forgets that. – Hans Passant Dec 14 '15 at 08:40
  • We've conducted experiments with one program repeatedly trying to read a file via a fixed pathname (File.ReadAllBytes) and another program concurrently repeatedly trying to rename and recreate that file (File.Move; File.WriteAllBytes). The latter process will occasionally throw an exception on File.Move if the other process is reading the file. – Rafe Dec 14 '15 at 22:25
  • Disable anti-malware and try again. If it the free crap (Avast, AVG) then get rid of it completely. – Hans Passant Dec 14 '15 at 22:52
  • I don't think AVG is involved. I've edited my question to include sample code showing the problem (you can run this directly in LINQPad if you add System.Threading.Tasks to the list of imported namespaces). – Rafe Dec 14 '15 at 23:12
  • You might need FileShare.Delete as well as FileShare.ReadWrite. – Harry Johnston Dec 15 '15 at 01:14
  • But failing that, you might be able to use shadow volumes as a (hideous) workaround. See [vssadmin](https://technet.microsoft.com/en-us/library/cc754968.aspx), for example. – Harry Johnston Dec 15 '15 at 01:17
  • @HarryJohnston -- That did the trick! Many thanks indeed. – Rafe Dec 15 '15 at 01:47

2 Answers2

0

In order to move a file, the sharing mode must allow you to delete it. In this context, that means the FileShare.Delete option.

Harry Johnston
  • 35,639
  • 6
  • 68
  • 158
  • Unfortunately, it seems there's something else to this story. The "unfixable" process in our case is a Delphi program which uses a library procedure called RenameFile. In my C# test program for the "reader" side, even if I open the file with new FileStream(filePath, FileMode.Open, FileAccess.Read, FileShare.Delete | FileShare.ReadWrite) and hold that file handle while the Delphi program calls RenameFile, the Delphi program aborts. I can only assume the Delphi code is asking for "super exclusive file access"! – Rafe Dec 16 '15 at 00:30
  • Note that using FileShare.Delete does appear to work fine when using a C# "writer" program using File.Move. – Rafe Dec 16 '15 at 00:31
  • Yes, if the other program is requesting exclusive access, and you already have read access, then your sharing mode becomes irrelevant. – Harry Johnston Dec 16 '15 at 03:10
0

NTFS has an interesting feature called “volume shadow copy” that allows you to take snapshots of the HDD while the HDD is in use. While other apps write to the HDD, those changes will be stored in another HDD sectors, while the shadow copy will still contains the data at the time the shadow copy was created. The main downside for you — they are relatively slow to initialize, if by "realtime" you mean you need to query that data each second, then I am afraid this method will not work for you. This is however the most reliable and least distracting way. Here’s some guide.

Another method that might work for you. Each time you want to read your data:

  • Suspend the writer process. There’re several ways.
  • Find the latest data file
  • Call CreateHardLink API to link that latest data file to some other path on the same NTFS volume.
  • Resume the writer process.
  • Now read that latest file opening it using the alternative path.

This way, the writer process should be able to rename the file just fine, even while you’re reading the same data using the alternate path.

Community
  • 1
  • 1
Soonts
  • 20,079
  • 9
  • 57
  • 130
  • Locks apply to the file, not to the specific link used to reference it. So that won't work. (But it might be sufficient to suspend the writer process, read the file, then resume.) – Harry Johnston Dec 16 '15 at 03:08
  • Suspending the writer process seems like it would do the trick -- of course, whether that particular bun-fight can be won with the other stakeholders is a big question! – Rafe Dec 17 '15 at 05:10
  • Just want to say many thanks for everybody's advice, by the way. – Rafe Dec 17 '15 at 05:11
  • @Rafe Just keep in mind that when you’ve suspended the writer, it could be it was in the middle of writing the new file. You should detect this case somehow, and if the new file is incomplete, your reader should use the previous file instead. The detection is best done outside the file system, e.g. if the file is xml look for the closing tag, or if the format contains checksum verify that, or if the format contains length verify that. If you can’t find a way to detect integrity of your files, you need much more complex code to enumerate file handles currently opened by your writer process. – Soonts Dec 19 '15 at 11:35