Read a buffer with background parallel infinite loop C#

Question

I have to create a WPF App (.NET Framework) to create a program with user interface.

While the GUI is working, the program has to continue, with an infinite loop, to read in background a folder which is continuously filled with new txt files from an external program.

An example:

• External programm --> creates new txt files and puts them into a folder named "Buffer"
• My Programm --> starts in background an infinite loop that reads every single file and deletes them from "Buffer"

This process has to be controlled by the main GUI, but it doesn't have to stop the other processes.

I suppose that I have to use threads to parallelize the processes, but unfortunately I don't have a lot of experience with C#, could you help me by giving some suggestion please?

Answer 1

I have been in your shoes, before. I cannot share the actual code (of course) but this is what I did:

• Separate "Watching a Folder" at least into its own class (if not sub-project).
• Separate "Process a file" at least into its own class(es).

For the Folder-Part:

1. I used FileSystemWatcher to get notified of new files present in a folder. All the EventHandler did was to put the filepath into a ConcurrentDictionary<string,DateTime>, storing "last seen" along with the path as key.

2. A timer that ran every (I think it was) 5 seconds filtered the Dictionary for all entries older than 2.5 seconds (half interval). This was to make sure, the file is not in transit anymore. Not really a "100% solution" but it worked well enough.

   The selected files (path strings) were set into a "blacklist" to be ignored in following runs and also sent to a Dataflow pipeline.

Then Processing:

3. a. The data flow pipeline did all the file processing and finally signalled fnished process, so the file could be moved to a "done" folder and removed from blacklist.

   b. Files that caused exceptions were put in a "manual interaction needed" Folder along with a corresponding "filename.error" file that contained exception message and stacktrace and was removed from blacklist. Mind that you might have to handle duplicate filenames, there.

The only distinction was that I had to do this in a Console App, that ran continuously on a server. Later on I refactored it to be an actual Windows Service. You however have to deal with WPF. The basic idea should stay the same, though. All you'll probably need is a proper way to display the process and interact with it. I strongly recommend to follow the MVVM pattern for that.

I also strongly recommend to adopt one of the beautiful logging frameworks out there.

Answer 2

If you really need an infinite loop doing some background work on files, you are not required to use threads explicitly with modern .NET. You may use TPL and launch a background task that runs the infinite loop once your app starts up. In the infinite loop, you should periodically check the cancellation token to enable graceful completion of the background task.

I think it makes sence to refer to the TPL documentation: https://learn.microsoft.com/en-US/dotnet/standard/parallel-programming/task-parallel-library-tpl

High-level code sample:

// Start your background task
void StartBackgroundWorker()
{
  // _cts is a data members of some object that encapsulates the background activity
  _cts = new CancellationTokenSource();
  System.Threading.Tasks.Task.Run(() => TaskEntryPoint(cts.Token));
}

// Stop your background task
void StopBackgroundWorker()
{
  _cts.Cancel();
  _cts.Dispose();
}

// In the background task, check for cancellation
void TaskEntryPoint(CancellationToken token)
{
  while (!token.IsCancellationRequested)
  {
    // complete a chunk of your background processing work
    // and check whether cancellation was requested by the 
    // owner of the background worker
  }
}