3

I have a large file(60mb) and I am reading the file into a string and Iam returning that string to another method.

Now when I am reading the file into a string its giving System out of memory exception.

Is there a way to read file in parts and append it to the string? If not is there a way around this?

static public string Serialize()
{
     string returnValue;
     System.IO.FileInfo file1 = new FileInfo(@"c:\file.txt");
     returnValue = System.IO.File.ReadAllText(file1.ToString());
}
RichardOD
  • 28,883
  • 9
  • 61
  • 81
shanthiram
  • 219
  • 1
  • 5
  • 14
  • 1
    see a similar question: http://stackoverflow.com/questions/247066/reading-and-parsing-files-in-net-performance-for-hire – Cleiton Sep 02 '09 at 20:17
  • just a nitpick: your method should read: Deserialize() Serializing is saving, deserializing loading. – Dabblernl Sep 02 '09 at 21:03

3 Answers3

6

How do you read the file right now ? You could use the StreamReader class, and read the file line by line (ReadLine method).
You could also read a specified amount of bytes from the file on each read operation (Read method)

Frederik Gheysels
  • 56,135
  • 11
  • 101
  • 154
  • The problem here is, as I am a beginner, I have to save that string into another file with encryption. – shanthiram Sep 02 '09 at 21:03
  • @shanthiram- then look at somethng like CryptoStream. Either way you will want to use a Stream. http://msdn.microsoft.com/en-us/library/system.security.cryptography.cryptostream.aspx – RichardOD Sep 02 '09 at 21:12
3

Yes- it's called streaming. Have a look at the StreamReader Class. Though I'm not sure why you want 1 60MB in one string. Probably best to deal with it a little at a time if possible (possibly in your scenario on a line by line basis?).

Instead of ReadAllText look at OpenRead and passing the returned FileStream into the constructor of a StreamReader, have a look at doing something along these lines if possible:

   using (FileStream fs = File.OpenRead("c:\theFile.text"))
   using (StreamReader sr = new StreamReader(fs))
   {
      string oneLine = sr.ReadLine();
   }
RichardOD
  • 28,883
  • 9
  • 61
  • 81
  • I have tried this and its still giving the System out of memory exception. – shanthiram Sep 02 '09 at 21:01
  • OK- then my guess if you are still loading it all into memory in one string. That's a bad idea. If you are doing encryption as you say you are, then look at http://msdn.microsoft.com/en-us/library/system.security.cryptography.cryptostream.aspx. – RichardOD Sep 02 '09 at 21:13
1

even if you read it line by line (or in parts by streaming), you will run out of memory as you are appending it to a single string. is compressing it along the way an option? if not, i'd probably up the maxHeap for the JVM to 512MB or similar.

cw22
  • 188
  • 5