Read a very large files in parallel C#

Question

I have more than 20 files, each of them contain almost 1 million lines (5 Gigabyte), I need to speed up the reading process, so I'm trying to read those files in parallel, but it takes longer time than reading them sequentially. is there any way to read a very large files in parallel?

 Parallel.ForEach(sourceFilesList, filePath =>
 {
     if (!string.IsNullOrEmpty(filePath) && File.Exists(filePath))
     {
          StreamReader str = new StreamReader(filePath);
          while (!str.EndOfStream)
          {
              var temporaryObj = new object();
              string line = str.ReadLine();
              // process line here 
          }
     }
});

How did you determine that it's not what you're doing in the processing that is the problem? — jmcilhinney, Mar 28 '18 at 06:06
and i see you are new to this site...So, Welcome to StackOverflow...If you do some research first,it'll be easy for you to fix ur issues fasster ..[Read this](https://stackoverflow.com/questions/36420013/open-and-read-thousands-of-files-as-fast-as-possible) — Software Dev, Mar 28 '18 at 06:08
This matter is covered regularly in questions and answers on SO, eg https://stackoverflow.com/questions/1367689/interleaved-parallel-file-read-slower-than-sequential-read. Do some reading around. — High Performance Mark, Mar 28 '18 at 06:09
This is most likely IO bottleneck... what you can do is benchmark your code and pinpoint where the bottleneck is. What do you do after reading each line? Memory could also be an issue if you don't dispose those quickly enough. — Rosdi Kasim, Mar 28 '18 at 06:10
the processing it's very simple, it takes less than 15 millisecond to process 1 line. — user7157121, Mar 28 '18 at 06:11
@user7157121Firstly, try to remove conditions in the loop and instead of ReadLine and while loop use ReadToEnd. Also, it's better to post your real code — Mahdi Farhani, Mar 28 '18 at 06:14

score 3 · Accepted Answer · answered Mar 28 '18 at 06:13

3

Its better to use buffered reader for huge files. something like this will help.

using (FileStream fs = File.Open(path, FileMode.Open, FileAccess.Read, 
FileShare.ReadWrite))
using (BufferedStream bs = new BufferedStream(fs))
using (StreamReader sr = new StreamReader(bs))
{
    string line;
    while ((line = sr.ReadLine()) != null)
    {

    }
}

Why BufferedStream is faster

A buffer is a block of bytes in memory used to cache data, thereby reducing the number of calls to the operating system. Buffers improve read and write performance. A buffer can be used for either reading or writing, but never both simultaneously. The Read and Write methods of BufferedStream automatically maintain the buffer.

answered Mar 28 '18 at 06:13

shrot

145
1
9

3

`FileStream` is already buffered, there's even a hint in the [reference source](https://referencesource.microsoft.com/#mscorlib/system/io/bufferedstream.cs,110) that it should not be used as a source for a `BufferedStream`. – Dirk Mar 28 '18 at 06:23
Thanks i wasn't aware of it. well then we can directly feed FS to SR. – shrot Mar 28 '18 at 06:27
As @Dirk , said , `FileStream` class is already buffered ... :) – Software Dev Mar 28 '18 at 06:32

score 1 · Answer 2 · edited Mar 28 '18 at 06:25

Its IO operation , suggestion is to make use of Async/Await like as below (mostly make use of ReadAsync function which helps to do read it asynchronous), Async/Await makes use of you Machine Physical Core's efficiently.

public void ReadFiles()
{
  List<string> paths = new List<string>(){"path1", "path2", "path3"};
  foreach(string path in Paths)
  {
      await ProcessRead(path);
  }
}

public async void ProcessRead(filePath)
{
    if (File.Exists(filePath) == false)
    {
        Debug.WriteLine("file not found: " + filePath);
    }
    else
    {
        try
        {
            string text = await ReadTextAsync(filePath);
            Debug.WriteLine(text);
        }
        catch (Exception ex)
        {
            Debug.WriteLine(ex.Message);
        }
    }
}

private async Task<string> ReadTextAsync(string filePath)
{
    using (FileStream sourceStream = new FileStream(filePath,
        FileMode.Open, FileAccess.Read, FileShare.Read,
        bufferSize: 4096, useAsync: true))
    {
        StringBuilder sb = new StringBuilder();

        byte[] buffer = new byte[0x1000];
        int numRead;
        while ((numRead = await sourceStream.ReadAsync(buffer, 0, buffer.Length)) != 0)
        {
            string text = Encoding.Unicode.GetString(buffer, 0, numRead);
            sb.Append(text);
        }

        return sb.ToString();
    }
}

Code is taken from MSDN : Using Async for File Access (C# and Visual Basic)

@zackraiyan - i tried but is showed me rejected by community , can you please do it again i will accept it asap — Pranay Rana, Mar 28 '18 at 06:23

Read a very large files in parallel C#

2 Answers2