C# Performance Comparison: Async vs Non-Async Text File IO operation via: Read\WriteAllLines vs Read\WriteAllLinesAsync

Question

While I was dealing with relatively large text files I noticed something strange. Async Reading and writing is actually slower than the non-async reading:

E,g, executing this dummy code:

 var res1 = File.WriteAllLinesAsync(string.Format(@"C:\Projects\DelMee\file{0}.txt", i), lines);
 var res2 = File.WriteAllLinesAsync(string.Format(@"C:\Projects\DelMee\file{0}_bck.txt", i), lines);

 await res1;
 await res2;

is actually much slower than

  File.WriteAllLines(string.Format(@"C:\Projects\DelMee\file{0}.txt", i), lines);
  File.WriteAllLines(string.Format(@"C:\Projects\DelMee\file{0}_bck.txt", i), lines);

In theory the first approach should be faster, because the second writing should be stared before the first one is completed. The performance difference is about 100% for 15 ~25MBs files (10 vs 20 seconds).

I noticed the same behavior for ReadAllLines and ReadAllLinesAsync.

Update:0 The main idea is to have all files processed after the completion of the function of TestFileWriteXXX functions. Therefore

Task.WhenAll(allTasks1); // Without await is not a valid option

Update:1 I added read write using threads and it showed 50% improvement. Here is the complete example:

Update:2 I updated the code in order to eliminate the buffer generation overhead

        const int MaxAttempts = 5;
        static void Main(string[] args)
        {
            TestFileWrite();
            TestFileWriteViaThread();
            TestFileWriteAsync();
            Console.ReadLine();
        }

        private static void TestFileWrite()
        {
            Clear();
            Stopwatch stopWatch = new Stopwatch();
            stopWatch.Start();

            Console.WriteLine( "Begin TestFileWrite");

            for (int i = 0; i < MaxAttempts; ++i)
            {
                TestFileWriteInt(i);
            }

            TimeSpan ts = stopWatch.Elapsed;
            string elapsedTime = String.Format("{0:00}:{1:00}:{2:00}.{3:00}", ts.Hours, ts.Minutes, ts.Seconds, ts.Milliseconds / 10);
            Console.WriteLine("TestFileWrite took: " + elapsedTime);
        }

        private static void TestFileWriteViaThread()
        {
            Clear();
            Stopwatch stopWatch = new Stopwatch();
            stopWatch.Start();

            Console.WriteLine("Begin TestFileWriteViaThread");

            List<Thread> _threads = new List<Thread>();

            for (int i = 0; i < MaxAttempts; ++i)
            {
                var t = new Thread(TestFileWriteInt);
                t.Start(i);
                _threads.Add(t);
            }

            _threads.ForEach(T => T.Join());

            TimeSpan ts = stopWatch.Elapsed;
            string elapsedTime = String.Format("{0:00}:{1:00}:{2:00}.{3:00}", ts.Hours, ts.Minutes, ts.Seconds, ts.Milliseconds / 10);
            Console.WriteLine("TestFileWriteViaThread took: " + elapsedTime);
        }

        private static void TestFileWriteInt(object oIndex)
        {
            int index = (int)oIndex;
            List<string> lines = GenerateLines(index);

            File.WriteAllLines(string.Format(@"C:\Projects\DelMee\file{0}.txt", index), lines);
            File.WriteAllLines(string.Format(@"F:\Projects\DelMee\file{0}_bck.txt", index), lines);

            var text = File.ReadAllLines(string.Format(@"C:\Projects\DelMee\file{0}.txt", index));
            var text1 = File.ReadAllLines(string.Format(@"C:\Projects\DelMee\file{0}.txt", index));

            //File.WriteAllLines(string.Format(@"C:\Projects\DelMee\file_test{0}.txt", index), text1);
        }

        private static  async void TestFileWriteAsync()
        {
            Clear();

            Console.WriteLine("Begin TestFileWriteAsync ");
            Stopwatch stopWatch = new Stopwatch();
            stopWatch.Start();

            for (int i = 0; i < MaxAttempts; ++i)
            {
                List<string> lines = GenerateLines(i);
                var allTasks = new List<Task>();

                allTasks.Add(File.WriteAllLinesAsync(string.Format(@"C:\Projects\DelMee\file{0}.txt", i), lines));
                allTasks.Add(File.WriteAllLinesAsync(string.Format(@"F:\Projects\DelMee\file{0}_bck.txt", i), lines));

                await Task.WhenAll(allTasks);

                var allTasks1 = new List<Task<string[]>>();
                allTasks1.Add(File.ReadAllLinesAsync(string.Format(@"C:\Projects\DelMee\file{0}.txt", i)));
                allTasks1.Add(File.ReadAllLinesAsync(string.Format(@"C:\Projects\DelMee\file{0}.txt", i)));

                await Task.WhenAll(allTasks1);

//                await File.WriteAllLinesAsync(string.Format(@"C:\Projects\DelMee\file_test{0}.txt", i), allTasks1[0].Result);
            }

            stopWatch.Stop();

            TimeSpan ts = stopWatch.Elapsed;
            string elapsedTime = String.Format("{0:00}:{1:00}:{2:00}.{3:00}", ts.Hours, ts.Minutes, ts.Seconds, ts.Milliseconds / 10);
            Console.WriteLine("TestFileWriteAsync took: " + elapsedTime);
        }

        private static void Clear()
        {
            for (int i = 0; i < 15; ++i)
            {
                System.IO.File.Delete(string.Format(@"C:\Projects\DelMee\file{0}.txt", i));
                System.IO.File.Delete(string.Format(@"F:\Projects\DelMee\file{0}_bck.txt", i));
            }
        }


        static string buffer = new string('a', 25 * 1024 * 1024);
        private static List<string> GenerateLines(int i)
        {
            return new List<string>() { buffer };
        }

And the results are:

TestFileWrite took: 00:00:03.50

TestFileWriteViaThread took: 00:00:01.63

TestFileWriteAsync took: 00:00:06.78

8 Code CPU/ C and F are 2 different SSD drive 850 EVOs on 2 different SATAs.

Update:3 - Conclusion Looks like the File.WriteAllLinesAsync does handle well the scenario when we want to flush large amount of data. As it was pointed out from the answers bellow it is better to use FileStream directly. But still async is slower than sequential access.

But for now the fastest approach still remains if you use multi threading.

Answer 1

I think this is a known issue. If you Google, you will see heaps similar posts.

Eg https://github.com/dotnet/runtime/issues/23196

If fast is the requirement for single IO operation, you should always use sync IO as well as sync methods.

Write*Async methods internally opens file stream in asynchronous IO mode, which has overhead cost compared to sync IO.

https://learn.microsoft.com/en-us/windows/win32/fileio/synchronous-and-asynchronous-i-o

However, for relatively fast I/O operations, the overhead of processing kernel I/O requests and kernel signals may make asynchronous I/O less beneficial, particularly if many fast I/O operations need to be made. In this case, synchronous I/O would be better.

Also asynchronous methods in FileStream and StreamWriter may have issues with small buffer size. The default buffer size for writing to file stream is 4KB, which is way too small than file size (25MB to 50MB). Despite the buffer size seems working for sync methods fine, but it exaggerates the overhead cost incurred by async methods.

See this line, the method yields thread every time when buffer is full. If you are writing a 25MB file with the default 4096 bytes buffer, this happens 6400 times.

To optimize that, if the entire file is in memory, the buffer size could be set to the file size to reduce context switching and synchronization between each write and flush.

If you open FileStream and StreamWriter with different buffer sizes in your code and run the tests for Write and WriteAsync, you will see difference. If the buffer size is identical to the file size, the difference between sync and async methods is very small.

For example:

// 4KB buffer sync stream
using (var stream = new FileStream(
    path, FileMode.Create, FileAccess.Write, FileShare.Read, 
    4096, FileOptions.SequentialScan))
{
    using (var writer = new StreamWriter(stream, Encoding.UTF8))
    {
        writer.Write(str25mb);
    }
}

// 25MB buffer sync stream
using (var stream = new FileStream(
    path, FileMode.Create, FileAccess.Write, FileShare.Read, 
    25 * 1024 * 1024, FileOptions.SequentialScan))
{
    using (var writer = new StreamWriter(stream, Encoding.UTF8))
    {
        writer.Write(str25mb);
    }
}

// 4KB buffer async stream
using (var stream = new FileStream(
    path,
    FileMode.Create, FileAccess.Write, FileShare.Read,
    4096, FileOptions.Asynchronous | FileOptions.SequentialScan))
using (var writer = new StreamWriter(stream, Encoding.UTF8))
{
    await writer.WriteAsync(str25mb);
}

// 25MB buffer async stream
using (var stream = new FileStream(
    path,
    FileMode.Create, FileAccess.Write, FileShare.Read,
    25 * 1024 * 1024, FileOptions.Asynchronous | FileOptions.SequentialScan))
using (var writer = new StreamWriter(stream, Encoding.UTF8))
{
    await writer.WriteAsync(str25mb);
}

Result (I run each test 10 times) was:

TestFileWriteWithLargeBuffer took: 00:00:00.9291647
TestFileWriteWithLargeBufferAsync took: 00:00:01.1950127
TestFileWrite took: 00:00:01.5251026
TestFileWriteAsync took: 00:00:03.6913877

Answer 2

I was wrong with my first answer since not used wait for Task.WhenAll when tried to remove async from the TestFileWriteAsync.

I've fixed test and it shows that File.Write*Async is really slower.

Begin TestFileWriteAsync
TestFileWriteAsync took: 00:00:13.7128699
Begin TestFileWrite
TestFileWrite took: 00:00:01.5734895
Begin TestFileWriteViaThread
TestFileWriteViaThread took: 00:00:00.8322218

Please excuse me

PS I've checked the Async methods source code.

Looks like File.WriteAllLinesAsync and File.WriteAllTextAsync uses the same InternalWriteAllTextAsync that copies the part of the original buffer one more time

buffer = ArrayPool<char>.Shared.Rent(DefaultBufferSize);
int count = contents.Length;
int index = 0;
while (index < count)
{
 int batchSize = Math.Min(DefaultBufferSize, count - index);
 contents.CopyTo(index, buffer, 0, batchSize);
#if MS_IO_REDIST
 await sw.WriteAsync(buffer, 0, batchSize).ConfigureAwait(false);
#else
 await sw.WriteAsync(new ReadOnlyMemory<char>(buffer, 0, batchSize), cancellationToken).ConfigureAwait(false);
#endif

The contents.CopyTo(index, buffer, 0, batchSize); is the line that copies the part of the original data buffer.

You can try with File.WriteAllBytesAsync it takes data buffer "as is" and does not execute additional copy operation:

Begin TestFileWriteAsync
TestFileWriteAsync took: 00:00:00.7741439
Begin TestFileWrite
TestFileWrite took: 00:00:00.5772008
Begin TestFileWriteViaThread
TestFileWriteViaThread took: 00:00:00.4457552

WriteAllBytesAsync test source code