7

Possible Duplicate:
How to read a large (1 GB) txt file in .NET?

What is the optimal way in C# to read an file, replace some strings and write in another new file? I need to do this with very large files like 8GB or 25GB.

Community
  • 1
  • 1
iLevi
  • 936
  • 4
  • 10
  • 26

3 Answers3

11

There isn't much you can optimize about the I/O, most of the optimization should be on the string comparison to determine if the string should be replaced or not, basically you should do this

protected void ReplaceFile(string FilePath, string NewFilePath)
{
   using (StreamReader vReader = new StreamReader(FilePath))
   {
      using (StreamWriter vWriter = new StreamWriter(NewFilePath))
      {
         int vLineNumber = 0;
         while (!vReader.EndOfStream)
         {
            string vLine = vReader.ReadLine();
            vWriter.WriteLine(ReplaceLine(vLine, vLineNumber++));
         }
      }
   }
}
protected string ReplaceLine(string Line, int LineNumber )
{
   //Do your string replacement and 
   //return either the original string or the modified one
   return Line;
}

What is your criteria to find and replace a string?

oamilkar
  • 326
  • 3
  • 5
  • If you don't need the line number, you should remove it from the ReplaceLine() declaration and from the calling to that function – oamilkar Apr 13 '12 at 23:05
  • Great answer. Shouldn't cause an exception from reading too much into memory per line. – Techgration Oct 29 '15 at 22:03
2

I did 200GB files before. Please use StreamReader and StreamWriter.

StreamReader

StreamWriter

1

Are there lines that are not overly large? If so, you can read in the file line by line, do the replace on that line, and then write out that line to the new file. Since it's streamed very little memory is required.

Servy
  • 202,030
  • 26
  • 332
  • 449