Write to a file from multiple threads asynchronously c#

Question

Here is my situation. I would like to make writing to the file system as efficient as possible in my application. The app is multi-threaded and each thread can possibly write to the same file. Is there a way that I can write to the file asynchronously from each thread without having the writes in the different threads bang heads together, so to speak?

I'm using C# and .NET 3.5, and I do have the Reactive Extensions installed as well.

Answer 1

For those who prefer code, I am using following to do remote logging from web apps...

public static class LoggingExtensions
{
    static ReaderWriterLock locker = new ReaderWriterLock();
    public static void WriteDebug(this string text)
    {
        try
        {
            locker.AcquireWriterLock(int.MaxValue); //You might wanna change timeout value 
            System.IO.File.AppendAllLines(Path.Combine(Path.GetDirectoryName(System.Reflection.Assembly.GetExecutingAssembly().GetName().CodeBase).Replace("file:\\", ""), "debug.txt"), new[] { text });
        }
        finally
        {
            locker.ReleaseWriterLock();
        }
    }
}

Hope this saves you some time

Answer 2

Have a look at Asynchronous I/O. This will free up the cpu to continue with other tasks.
Combine with ReaderWriterLock as @Jack B Nimble mentioned

If by

writing to the file system as efficient as possible

you mean making the actual file I/O as fast as possible you are going to have a hard time speeding it up much, disk is just physically slower. Maybe SSD's?

Answer 3

What I would do is have separate worker thread(s) dedicated to the task of writing files out. When one of your other threads needs to write some data out, it should call a function to add the data to an ArrayList (or some other container/class). Inside this function, there should be a lock statement at the top to prevent more than one thread from executing simultaneously. After adding the reference to the ArrayList it returns and continues on with its chores. There are a couple of ways to handle the writing thread(s). Probably the simplest is to simply put it into an infinite loop with a sleep statement at the end so that it does not chew up your cpu(s). Another way is to use thread primitives and go into a wait state when there is no more data to be written out. This method implies that you would have to activate the thread with something like the ManualResetEvent.Set method.

There are many different ways to read in and write out files in .NET. I have written a benchmark program and give the results in my blog:

http://designingefficientsoftware.wordpress.com/2011/03/03/efficient-file-io-from-csharp

I would recommend using the Windows ReadFile and WriteFile methods if you need performance. Avoid any of the asynchronous methods since my benchmark results show that you get better performance with synchronous I/O methods.

Answer 4

While thread based locks can solve this, there is a manner which works across threads, but is probably best used when you have multiple processes writing to the end of a single file.

To get this behavior across processes (or threads too), specify that you want atomic append writes to the operating system when the OS file handles are created. This is done by specifying O_APPEND under Posix(Linux,Unix), and FILE_APPEND_DATA under Windows.

In C# you don't call the OS 'open', or 'CreateFile' system calls directly, but there are ways to get this result.

I asked how to do this under Windows a while ago, and got two good answers here: How can I do an atomic write/append in C#, or how do I get files opened with the FILE_APPEND_DATA flag?

Basically, you can use FileStream() or PInvoke, I would suggest FileStream() over PInvoke for obvious reasons.

You can use constructor arguments to FileStream() to specify asynchronous file I/O in addition to the FileSystemRights.AppendData flag, which should give you both async I/O and atomic append writes to a file.

Warning: Some OSes have limits on the maximum number of bytes that can be atomically written this way, and exceeding that threshold will remove the OS promise of atomicity.

Because of this last gotcha, I would recommend staying with lock() style contention management when trying to address your problem within a single process.

Answer 5

Use Reader / Writer locks to access the file stream.

Answer 6

Save to log with Queue and multiple threads (.Net Core 2.2 linux sample - tested)

using System;
using System.Collections;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Text.RegularExpressions;
// add
using System.Security.Cryptography.X509Certificates;
using System.Threading;
using System.Net;
using System.Net.Sockets;
using System.Net.Security;
using System.Security.Authentication;
using System.IO;
using System.Timers;

namespace LogToFile
{
    class Program
    {
        public static Logger logger = new Logger("debug.log");

        static void Main(string[] args)
        {
            Console.WriteLine("Hello World!");

            logger.add("[001][LOGGER STARTING]");

            Thread t0 = new Thread(() => DoWork("t0"));
            t0.Start();

            Thread t1 = new Thread(() => DoWork("t1"));
            t1.Start();

            Thread t2 = new Thread(() => DoWork("t2"));
            t2.Start();

            Thread ts = new Thread(() => SaveWork());
            ts.Start();
        }

        public static void DoWork(string nr){
            while(true){
                logger.add("Hello from worker .... number " + nr);
                Thread.Sleep(300);
            }
        }

        public static void SaveWork(){
            while(true){
                logger.saveNow();
                Thread.Sleep(50);
            }
        }
    }

    class Logger
    {
        // Queue import: 
        // using System.Collections
        public Queue logs = new Queue();
        public string path = "debug.log";

        public Logger(string path){
            this.path = path;
        }

        public void add(string t){
            this.logs.Enqueue("[" + currTime() +"] " + t);
        }

        public void saveNow(){
            if(this.logs.Count > 0){
                // Get from queue
                string err = (string) this.logs.Dequeue();
                // Save to logs
                saveToFile(err, this.path);
            }
        }

        public bool saveToFile(string text, string path)
        {
            try{
                // string docPath = Environment.GetFolderPath(Environment.SpecialFolder.MyDocuments);
                // text = text + Environment.NewLine;
                using (StreamWriter sw = File.AppendText(path))
                {
                    sw.WriteLine(text);
                    sw.Close();
                }
            }catch(Exception e){
                // return to queue
                this.logs.Enqueue(text + "[SAVE_ERR]");
                return false;
            }
            return true;
        }

        public String currTime(){
            DateTime d = DateTime.UtcNow.ToLocalTime();
            return d.ToString("yyyy-MM-dd hh:mm:ss");
        }
    }
}

Compile (Save to: LogToFile/Program.cs):

dotnet new console -o LogToFile
cd LogToFile
dotnet build
dotnet run

Stop app CTRL+C and see log file

cat debug.log

Answer 7

You can use events for logger:

using System;
using System.IO;
using System.Threading;
using System.Threading.Tasks;

namespace EventLogger
{
    class Program
    {
        static void Main(string[] args)
        {                   
            // Event handler
            LogData ld = new LogData();
            // Logger
            Logger lo = new Logger();                    
            // Subscribe to event
            ld.MyEvent += lo.OnMyEvent;     
            // Thread loop
            int cnt = 1;
            while(cnt < 5){         
                Thread t = new Thread(() => RunMe(cnt, ld));
                t.Start();
                cnt++;
            }
            Console.WriteLine("While end");
        }

        // Thread worker
        public static void RunMe(int cnt, LogData ld){
            int nr = 0;
            while(true){
                nr++;
                // Add user and fire event
                ld.AddToLog(new User(){Name = "Log to file Thread" + cnt + " Count " + nr, Email = "em@em.xx"});
                Thread.Sleep(1);
            }
        }
    }

    class LogData
    {
        public delegate void MyEventHandler(object o, User u);
        public event MyEventHandler MyEvent;

        protected virtual void OnEvent(User u)
        {
            if(MyEvent != null){
                MyEvent(this, u);
            }

        }

        // Wywołaj
        public void AddToLog(User u){
            Console.WriteLine("Add to log.");

            // Odpal event
            OnEvent(u);

            Console.WriteLine("Added.");
        }
    }

    class User
    {
        public string Name = "";
        public string Email =  "";
    }

    class Logger
    {
        // Catch event
        public void OnMyEvent(object o, User u){
            try{
                Console.WriteLine("Added to file log! " + u.Name + " " + u.Email);
                File.AppendAllText(@"event.log", "Added to file log! " + u.Name + " " + u.Email+"\r\n");
            }catch(Exception e){
                Console.WriteLine("Error file log " + e);
            }
        }
    }
}