System.OutofMemoryException throw while doing GZipStream Compression

Question

I am working in win forms. Getting errors while doing following operation. It shows me System.OutOfMemoryException error when i try to run the operation around 2-3 times continuously. Seems .NET is not able to free the resouces used in operation. The file i am using for operation is quite big, around more than 500 MB.

My sample code is as below. Please help me how to resolve the error.

try
{
   using (FileStream target = new FileStream(strCompressedFileName, FileMode.Create, FileAccess.Write))
   using (GZipStream alg = new GZipStream(target, CompressionMode.Compress))
   {
       byte[] data = File.ReadAllBytes(strFileToBeCompressed);
       alg.Write(data, 0, data.Length);
       alg.Flush();
       data = null;
    }
}
catch (Exception ex)
{
    MessageBox.Show(ex.ToString());
}

Instead of `ReadAllBytes`, open source file and use its `Read` method in a loop — L.B, Aug 27 '12 at 13:39
`ReadAllBytes` is the killer - this gets all your more than 500 MB to RAM — Renatas M., Aug 27 '12 at 13:39
byte[] data = File.ReadAllBytes(strFileToBeCompressed); The line where OOM comes — amit patel, Aug 27 '12 at 13:41
The accepted answer to your previous question still holds true http://stackoverflow.com/questions/12038694/reading-large-file-using-byte-gives-error — Security Hound, Aug 27 '12 at 13:56

score 7 · Answer 1 · answered Aug 27 '12 at 13:43

7

Replace ReadAllBytes with Stream.CopyTo

using (FileStream target = new FileStream(strCompressedFileName, FileMode.Create, FileAccess.Write))
using (GZipStream alg = new GZipStream(target, CompressionMode.Compress))
{
    using (var fileToRead = File.Open(.....))
    {
        fileToRead.CopyTo(alg);
    }
}

answered Aug 27 '12 at 13:43

L.B

114,136
19
178
224

1

That will automatically use a sensible buffer size, but if you want to specify a buffer size there is a CopyTo() overload to do so. It is not necessary IMO, just letting the OP know it's there. The default buffer size is just 4096. – Matthew Watson Aug 27 '12 at 13:46

score 6 · Accepted Answer · answered Aug 27 '12 at 13:43

6

A very rough example could be

// destFile - FileStream for destinationFile 
// srcFile - FileStream of sourceFile
using (GZipStream gz = new GZipStream(destFile, CompressionMode.Compress))
{
     byte[] src = new byte[1024];
     int count = sourceFile.Read(src, 0, 1024);
     while (count != 0)
     {
         gz.Write(src, 0, count );
         count = sourceFile.Read(src, 0, 1024);
     }
}
// flush, close, dispose ..

So basically I changed your ReadAllBytes to read only chunks of 1024 bytes.

answered Aug 27 '12 at 13:43

Pilgerstorfer Franz

8,303
3
41
54

Yes, but CopyTo is easier. Also, use a larger buffer, like `8*1024` . – H H Aug 27 '12 at 13:45
Of course you are right Henk, but I do think amit can adapt this on his own. @CopyTo thx for the hint! – Pilgerstorfer Franz Aug 27 '12 at 13:48

user854301 · Answer 3 · 2012-08-27T13:50:29.273

You can try to use this method to compress file MSDN link

    public static void Compress(FileInfo fileToCompress)
    {
        using (FileStream originalFileStream = fileToCompress.OpenRead())
        {
            using (FileStream compressedFileStream = File.Create(fileToCompress.FullName + ".gz"))
            {
               using (GZipStream compressionStream = new GZipStream(compressedFileStream, CompressionMode.Compress))
               {
                      originalFileStream.CopyTo(compressionStream);                                
               }
            }
        }
    }

usage:

string directoryPath = @"c:\users\public\reports";

DirectoryInfo directorySelected = new DirectoryInfo(directoryPath);

foreach (FileInfo fileToCompress in directorySelected.GetFiles())
{
    Compress(fileToCompress);
}

System.OutofMemoryException throw while doing GZipStream Compression

3 Answers3