FileSystemWatcher losing files in its queue

Question

I have written a FileSystemWatcher to call a pgm once for every file. But some of my files are lost. I tested the code with only 10-11 files. Deletion of a file is logged correctly, but not the creation. Some of the files are not logged. Is there maybe any problem in my TASK implementation? or is there any problem with Window Service?

 public static FileSystemWatcher m_Watcher;
        static BlockingCollection<string> blockingCollection = new BlockingCollection<string>();

protected override void OnStart(string[] args)
    {

        current_directory = Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location);
        //XmlDocument xml = new XmlDocument();
        try
        {
            strDir = ConfigurationManager.AppSettings["Directory"];
            fileMask = ConfigurationManager.AppSettings["FileMask"];
            strBatfile = ConfigurationManager.AppSettings["Batch"];
            strlog = ConfigurationManager.AppSettings["Log"];

            m_Watcher = new FileSystemWatcher();


            m_Watcher.Filter = fileMask;
            m_Watcher.Path = strDir + "\\";
            m_Watcher.NotifyFilter = NotifyFilters.LastAccess | NotifyFilters.LastWrite
                             | NotifyFilters.FileName | NotifyFilters.DirectoryName;




            m_Watcher.Created += new FileSystemEventHandler(OnCreated);

            m_Watcher.Deleted += new FileSystemEventHandler(OnDeleated);
            m_Watcher.Renamed += new RenamedEventHandler(OnRenamed);


            m_Watcher.EnableRaisingEvents = true;
        }
        catch (Exception exception)
        {
            CustomException.Write(CustomException.CreateExceptionString(exception.ToString()));
        }

    }
    public static void OnDeleated(object source, FileSystemEventArgs e)
    {
        try
        {

            Log.getLogger("File deleated- Filename :" + e.Name + " at timestamp : " + DateTime.Now.ToString(), strlog);
        }
        catch (Exception exception)
        {
            CustomException.Write(CustomException.CreateExceptionString(exception, e.Name));
        }
    }

    private static void OnCreated(object source, FileSystemEventArgs e)
    {

        var exceptions = new ConcurrentQueue<Exception>();

        try
        {

            Task.Factory.StartNew(() =>
            {
                try
                {

                    blockingCollection.Add(e.Name.ToString());

                }
                catch (Exception)
                {
                    throw;

                }

            });
            Task.Factory.StartNew(() =>
            {
                try
                {


                    foreach (string value in blockingCollection.GetConsumingEnumerable())
                    {
                        System.Diagnostics.Process.Start(Service1.strBatfile);
                        Log.getLogger("File Processed after executing batch:  Filename ->:" + value + " " + "Batch File Executed- > " + Service1.strBatfile + " at timestamp : " + DateTime.Now.ToString(), Service1.strlog);

                    }
                }
                catch (Exception)
                {
                    throw;
                }



            });


        }
        catch (AggregateException ae)
        {

            foreach (var ex in ae.InnerExceptions)
            {
                CustomException.Write(CustomException.CreateExceptionString(ex, e.Name));
            }
        }
        finally
        {
            m_Watcher.EnableRaisingEvents = true;
        }
    }

Answer 1

You are using way to many threads/Tasks to get a clear understanding how the code works. As you stated that you want to only process one file at a time you need just one Thread/Task that lives for the lifetime of the class (and I assume the application).

If I strip down your code to accomplish processing one file at a time whenever a file is dropped in a certain folder this could be one implementation.

Notice how I have one ConcurrentQueue and ONE method that reads that queue. I also use the method WaitForExit on the process instance to prevent running more than one process.

    static ConcurrentQueue<string> filenames = new ConcurrentQueue<string>();

    static void QueueHandler()
    {
        bool run = true;
        AppDomain.CurrentDomain.DomainUnload += (s, e) =>
        {
            run = false;
            filenames.Enqueue("stop");
        };
        while(run)
        {
            string filename;
            if (filenames.TryDequeue(out filename) && run)
            {
                var proc = new Process();
                proc.StartInfo.FileName = filename;
                proc.Start();
                proc.WaitForExit(); // this blocks until the process ends....

            }
        }
    }

Now we need a single Task/Thread that will run QueueHandler and our FileSystemWatcher:

protected override void OnStart(string[] args)
{
        // have our queue reader method started
        Task.Factory.StartNew(QueueHandler);

        var fsw = new FileSystemWatcher();
        fsw.Created += (o, e) =>
            {
                // add a file to the queue
                filenames.Enqueue(e.FullPath);
                // optionally add polling for missed files
                // http://stackoverflow.com/questions/239988/filesystemwatcher-vs-polling-to-watch-for-file-changes
            };

        fsw.Path = ConfigurationManager.AppSettings["Directory"];
        fsw.NotifyFilter = NotifyFilters.FileName;
        fsw.Filter = ConfigurationManager.AppSettings["FileMask"];

        fsw.EnableRaisingEvents = true;
}

This implementation will use at worst three threads: one main thread, one for the Created events of the FileSystemWatcher and one for the QueueHandler instead if your example code where new Tasks were started every time a new file was created in the folder the FileSystemWatcher was watching

Answer 2

You are starting two tasks in your OnCreated method, where the second task seems to depend on the output from the first task. However, there is no guarantee that the first task will have finished (or even started) when the second task executes.

You could group the two operations into a single task, which would then execute sequentially, or you could await the result of the first task.

There is also a lot of information missing from your code. It clearly isn't the 'real' code because OnDeleated [sic] is misspelled and wouldn't compile. We also can't see what your external process is or how you are attempting to pass the file list to it. There could be lots of problems there. Would it be possible to post the actual code?