2

I am rather new to C# and trying to determine best practice when it comes to file operations.

My application is using Parallel.ForEach to parse several large files. In the method, I am writing to a single file with StreamWriter but having to lock the object during that process. I anticipated a slow down with that methodology and knew that I would have to have a separate thread for file IO.

Which brings me to my question. Does C# already provide a namespace for non-blocking, thread-safe file IO?

Thanks!

user0000001
  • 2,092
  • 2
  • 20
  • 48
  • Possible duplicate of: http://stackoverflow.com/questions/19976449/is-file-readalltext-thread-safe – Tony_KiloPapaMikeGolf Dec 01 '16 at 21:53
  • You might get a speed boost if you delay writes and you write the data to files overtime instead of doing parallel writes. – Matías Fidemraizer Dec 01 '16 at 22:02
  • As far as I am aware, due to the nature of IO, it's not possible to have a truly parallel operation working on the same stream resource. – Abion47 Dec 01 '16 at 22:07
  • @Abion47 The question isn't asking for that. In fact, he specifically says that he's explicitly serializing it. – Servy Dec 01 '16 at 22:08
  • @Servy He is also asking if there is a non-blocking, thread-safe method to perform IO. I took that to mean he is looking for a method that will allow him to safely multithread his writes to `StreamWriter`. Perhaps I am mistaken in that assumption? – Abion47 Dec 01 '16 at 22:11
  • @Abion47 He wants an operation that's non-blocking (i.e. asynchronous) thread safe (meaning it's called from multiple threads at the same time) and that performs file IO. The actual file itself cannot be manipulated in parallel, but one can create an operation that can safely be called from multiple threads in parallel and that will synchronize them for you. – Servy Dec 01 '16 at 22:12

1 Answers1

3

.NET provides operations for asynchronous file IO, and it provides tools for synchronizing asynchronous operations. You have to combine the two tools yourself if you want both features though, not that that's particularly hard. You can create a simple queue to serialize asynchronous operations for you, as shown here. Once you have that you can share a single queue between all of your workers, having them each enqueue a call to WriteAsync to do your writing.

Note that you don't actually need a separate thread for the file writing at all. It can be asynchronous without needing a dedicated thread.

Community
  • 1
  • 1
Servy
  • 202,030
  • 26
  • 332
  • 449
  • I figured this would be my route. Thank you for the clarification! – user0000001 Dec 02 '16 at 18:03
  • Servy, I am just getting around to implementing this. I understand what a Semaphore is but I am trying to understand how I am going to use this in my particular use case. Do I create an `async Task` method and `Enqueue` that before I begin parsing the data? Then lock / unlock this queue when there is data to be written? I created minimal example https://dotnetfiddle.net/WKcS6H. Look at `TaskWrite`, is that the direction I want to go? – user0000001 Dec 05 '16 at 14:59
  • @user0000001 You simply need to create an `async` method that writes whatever you want out to a file (using the asynchronous operations that the .NET file operations expose) and enqueue calls to that method. You never need to actually touch the lock; the queue exists to encapsulate all of the behavior needed of ensuring that the methods run sequentially. – Servy Dec 05 '16 at 15:01