Wait until last file is copied with FileSystemWatcher created event

Question

I have a few python unit test files. I'm copying all those files to a directory, which a file system watcher is listening and firing create event. I want to run those tests after all files are copied. The files that are being copied look something like:

unittest1.py

unittest2.py

unittestxyz.py
.
.
.

unittestRunner.py

unittestRunner.py file imports all other unittest files and starts all unit tests as a part of a suite. If I copy files to the watched directory, Created event will fire that many times, in this case 4 times. So, somehow I want to 'wait' until all files are copied and then execute the unittestRunner script. Files are being copied programmatically, and watcher is running as a part of a Windows Service.

I have 2, quite yuck solutions, and they may not even be solutions:

one is to specifically wait for unittestRunner.py to be created:

 watcher.Created += new FileSystemEventHandler((sender, eventArgs) =>
                {
                    if (eventArgs.Name == "unittestRunner.py")
                    {
                      // Run Tests
                    }
                });

but this won't work if runner file gets copied before any other file.

Other is to keep track the number of files that have to be copied somewhere in an xml file, and in the evnet handler check if the current number of copied files are the same:

watcher.Created += new FileSystemEventHandler((sender, eventArgs) =>
                {
                    var totalCount = Directory.GetFiles(testFolder).Length;

                    var doc = new XmlDocument();
                    doc.Load("numberOfFilesToCopy.xml");

                    if (totalCount == doc.GetElementsByTagName("test-files")[0].ChildNodes.Count)
                    {
                        // Number is the same, all files copied, run tests
                    }
                }

Is it somehow possible to wait for all the locks of all the files have been released before starting tests?

Aren't you effectively asking for a piece of software that can predict the future? It somehow has to "know" whether yet another file will appear later (but close enough in time to be considered part of the same batch) — Damien_The_Unbeliever, Aug 12 '15 at 07:23
In a way, or just use something predefined, like an xml file which will store the number of files that are to be copied. That could work, but I'm not sure if that's the best solution to the problem. — Mefhisto1, Aug 12 '15 at 07:24
You can't know when the user will copy yet-another-file. There's also no "file closed" event. What you can do is wait for X seconds of no file system activity before you start processing the files. In an NTFS drive you could use its shadow copying capabilities to read even files that are still locked. You'd have to use a library like AlphaFS for that though — Panagiotis Kanavos, Aug 12 '15 at 07:25
@PanagiotisKanavos That is correct, however, I know the number of files that will be copied, since the copy-process is actually a build step as a part of a build, so when the build is run, I just copy all files that are in a specified folder, so in that respect, I can know the number of files that have to be copied — Mefhisto1, Aug 12 '15 at 07:30
Also, beware that the OS only has a limited buffer for file change events. If you do a lot of long running work in your change event, and more events come in quickly, that queue can overflow, dropping events or throwing an error. The best practice is to make your own queue of file system events and just populate that from the watcher's event, exiting as soon as possible. Then you can take your time processing your own internal queue without worrying about the OS'. — Bradley Uffner, Feb 23 '18 at 12:31

score 2 · Answer 1 · edited Feb 23 '18 at 12:37

You can check each file to see if they're accessible. Going from this SO post, you could do the following:

private bool WaitForFile(FileInfo file)
{
    var attemptCount = 0;

    while(true)
    {
        try
        {
            using(file.Open(FileMode.Open, FileAccess.ReadWrite, FileShare.None)) 
            { 
                return true; 
            }
        }
        catch (IOException)
        {
            //File isn't ready yet, so we need to keep on waiting until it is.
        }

        //We'll want to wait a bit between polls, if the file isn't ready.
        attemptCount++;

        if(attemptCount > 5)
        {
            break;
        }

        Thread.Sleep(1000);
    }

    return false;
}

Your idea about using an XML file to keep track of the number of expected files to be copied is also a good one. I recommend you have

var doc = new XmlDocument(); doc.Load("numberOfFilesToCopy.xml");

outside the method though so you don't have to load the same file repeatedly.

Also, beware that the OS only has a limited buffer for file change events. If you do a lot of long running work that blocks the thread in your change event, and more events come in quickly, that queue can overflow, dropping events or throwing an error. The best practice is to make your own queue of file system events and just populate that from the watcher's event, exiting as soon as possible. Then you can take your time processing your own internal queue without worrying about the OS'. It's generally a good place to use a Producer / Consumer Pattern. — Bradley Uffner, Feb 23 '18 at 12:33

score 2 · Answer 2 · answered Aug 26 '19 at 14:09

In a similar situation, I went this way:

Create a "Lock" File in the target directory. Nothing is startet, as long as this file is present.
Copy all needed files to target directory.
Delete "Lock" File.

The Program reacted on "Deleted" Event of the "Lock" file + Some Delay to ensure all concurrent access will be ended.

I even later on added a second lock file (for the other way), so the target dir won't be copied into while the run of the program was still in progress.

Wait until last file is copied with FileSystemWatcher created event

2 Answers2