C# Daemon for reading large amount of files

Question

I will start by giving some context first

Introduction

I am creating a program that will run continuously on a server. Part of the program is the ability to send automated mails to different representatives. The content of these mails will be automatically put in a folder, so the program needs to be notified when new content is ready and read the text that is in this file. However this is not a problem and I already have my solution for this.

The Problem

It is possible that there will be multiple files placed in the folder at the same time, here is when my program cannot handle it. I get the following exception :

System.IO.IOException: The process cannot access the file GENERATEDFILE because it is being used by another process.

As soon as the program tries to read a file that has been put in the folder,it will give me this exception, and I cannot wrap my head around why it does

What I have already

So I have a method that creates the FileSystemWatcher like this :

public void Watch(){
_watcher = new FileSystemWatcher();
_watcher.NotifyFilters =  NotifyFilters.LastAccess |
                          NotifyFilters.LastWrite |
                          NotifyFilters.FileName |
                          NotifyFilters.DirectoryName;
_watcher.Created += OnCreated

}

private void OnCreated(object sender, FileSystemEventArgs e){
//read the file that got created here and pass the content to an object that wille handle it further
}

So pretty easy, we create a FileSystemWatcher, we attach some NotifyFilters to it and the OnCreated event

I have tried the following

In the OnCreated method i have tried the following

1. File.ReadAllText(filePath) I would assume this was the standard way of reading files, however this does not work
2. using (StreamReader sr = new StreamReader(filePath)){ String line = sr.ReadToEnd();} From microsoft
3. I have tried Thread.Sleep(5000) to see if it needs some time, no effect
4. I have tried using the ReaderWriterLockSlim class that enters the readlock everytime i start reading, this solution sometimes works, but is not good enough to guarantee
5. I have tried putting each file in a list, then iterating over that list but still gives me the same exception

The Question

Is there any way to handle the event where multiple files get put in the folder at the same time?

Answer 1

How about catching the exception and setting a timer to retry later?

Answer 2

The Cause
File created event happens when the file is created, not when it is finished writing out.

A Solution
Consider adding a task for the processing logic.
When the file is created, the filename/path gets handed to a method that runs in a task that keeps retrying.

using System;
using System.Collections.Concurrent;
using System.IO;
using System.Threading.Tasks;
using Microsoft.VisualStudio.TestTools.UnitTesting;

namespace UnitTestProject1
{
    [TestClass]
    public class UnitTest1
    {
        private ConcurrentQueue<string> _filesProcessQueue;
        private FileSystemWatcher _watcher;

        [TestMethod]
        public void Watch()
        {
            _filesProcessQueue = new ConcurrentQueue<string>();
            _watcher = new FileSystemWatcher
            {
                NotifyFilter = NotifyFilters.LastAccess |
                               NotifyFilters.LastWrite |
                               NotifyFilters.FileName |
                               NotifyFilters.DirectoryName
            };
            _watcher.Created += OnCreated;
        }

        private void OnCreated(object sender, FileSystemEventArgs e)
        {
            // Create a task for reading the file after the lock is released
            Task.Run(() => ProcessFileQueue(e.FullPath));
        }

        private void ProcessFileQueue(string fullPath)
        {
            bool processSuccess = false;
            // TODO put terminator logic in here to stop retrying forever
            while (processSuccess == false)
            {
                // TODO try process the file and return success/fail
                processSuccess = true; // Continue to next file
                //processSuccess = false; // Retry the same file

                Task.Delay(TimeSpan.FromSeconds(5)); // Delay 5 seconds before retrying the file.
            }
        }
    }
}