0

There is a system who put files on a folder on a disk.

I'm writing an executable (c#) which take these files and send them into a database.

My executable can be started multiples times in the same time (in parallel) and I have a multithread problem with processing files.

Example:

  • There are 50 files in folder.
  • The executable 1 takes 10 files to process.
  • The executable 2 takes 10 files to process.

My questions are:

  • How can I be sure that my executable 2 don't take executable 1 files?
  • How can I lock the 10 files from executable 1?
  • How to make this process thread safe?
Uwe Keim
  • 39,551
  • 56
  • 175
  • 291
papoxe
  • 53
  • 4
  • 1
    You realise this will significantly slow you application down? – Liam Sep 12 '22 at 10:42
  • 1
    Is starting the application multiple times something you want to be able to do? And if so why? – Magnus Sep 12 '22 at 10:50
  • I have not realise, it will slow down the application. In fact, there are above 500 000 files in the folder and I want to launch it multiples times to increase speed of processing precisely. – papoxe Sep 12 '22 at 11:08

2 Answers2

3

Instead of using multiple processes, a more elegant solution would be to use one process with multiple threads.

You could for example have one thread that lists the files on the disk and adds them to a concurrentqueue, with multiple processing threads taking filenames from the queue and doing the processing. . Or use something like dataflow to essentially do the same thing. The absolutely simplestion option might be to create a list of all files, and use Plinq AsParallel to parallelize the processing.

Note that IO typically do not parallelize very well, so if "processing" mostly involve Reading and writing to disk, your gains might be less than expected.

If you insist on using multiple processes you could follow the same pattern by using an actor model, or otherwise having one management process handing out work to multiple worker processes.

I would probably not recommend relying only on file-locks, since it would be difficult to ensure that files are not processed twice.

JonasH
  • 28,608
  • 2
  • 10
  • 23
  • Thank's for your help, I will try this, instead of running app multiple times, I will use one process with multiple thread with PLinq ! – papoxe Sep 12 '22 at 11:27
1

It is better to read the list of all the files first, then divide between the threads. Otherwise, exception can be used

You can check if the file is in use or not.

 try
{
   using (Stream stream = new FileStream("File.txt", FileMode.Open))
   {
    // File ready for 
   }
} catch {
  //the file is in use.
}

Hope this helps!

Hossein Sabziani
  • 1
  • 2
  • 15
  • 20