0

I keep getting a very annoying OutOfMemory exception on the following code.

I'm zipping a lot of small files (PDF, each being 1.5mb approx).

At first I was getting the exception afer 25 files zipped, which doesn't seem like a massing archive.

Setting up the size of the ZipEntry somehow helped since now I manage to get up to 110 files zipped (I'm debugging under visual studio)

Here's my code, maybe there's something wrong with it.

Any help would be greatly appreciated.

Thanks

    public static MemoryStream Zip(Dictionary<string, byte[]> files)
    {
        var outputMemStream = new MemoryStream();

        var zipStream = new ZipOutputStream(outputMemStream);

        zipStream.SetLevel(9);
        foreach (var file in files)
        {
            zipStream.PutNextEntry(new ZipEntry(file.Key.FmtValidFileName())
                {
                    Size = file.Value.Length
                });
            zipStream.Write(file.Value, 0, file.Value.Length);
            zipStream.Flush();
        }           
        zipStream.Finish();
        outputMemStream.Position = 0;
        return outputMemStream;
    }
Etienne
  • 1,058
  • 11
  • 22
  • Every class in there that implements `IDisposible` needs to be put into a `using` block. If you have a Visual Studio version that supports it, you can turn on static code analysis so you get warnings if you forget to do it properly. – nvoigt Oct 17 '14 at 04:45
  • It doesn't help I'm afraid. The OutOfMemoryException happens within the loop on files. I can't put the outputMemStream inside a using because I'm using it as the return. I tried to use buffer instead of writing the file entirely but it doesn't help either, the problem is not the size of each file I'm zipping but the total size of all files. – Etienne Oct 17 '14 at 04:54
  • You should create a small, self contained example. For all we know, `FmtValidFileName` could consist of `throw new OutOfMemoryException();`. Prepare an example that people can copy and compile themselves using best practices and it will be a lot easier to help you. – nvoigt Oct 17 '14 at 05:00
  • FmtValidFileName is an extension method for a string to remove any fuzzy character from the filename that is being zipped so it's not the problem. Also I cannot give more than the code extract I've given, it's the smallest I got, I'm running it against a Dictionary files consisting of 160 records, each byte[] being approx 1434167 byte which is 1.3Mb more or less and it throws the OutOfMemory after approx a 100 files have been zipped. – Etienne Oct 17 '14 at 05:09

2 Answers2

2

As always, a concise-but-complete code example would go a long way toward helping you get good answers.

That said, you might want to consider using the (relatively new) System.IO.Compression.ZipArchive class in .NET. It is potentially less buggy and/or more reliable than third party libraries (though I admit SharpZipLib is reasonably well-respected :) ).

More important, you can instantiate a new ZipArchive object with the ZipArchiveMode.Create value, which will cause the compressed data to be written directly to the stream rather than being cached in-memory. In this mode, out-of-memory errors should be non-existent, no matter how much data or how many archive items you're trying to create.

EDIT: One more thing: to ensure completely against out-of-memory problems, make sure that whatever .zip implementation you are using, you write directly to the disk. Writing to a temporary in-memory MemoryStream will of course impose limitations on your process that otherwise need not occur.

Bob Kaufman
  • 12,864
  • 16
  • 78
  • 107
Peter Duniho
  • 68,759
  • 7
  • 102
  • 136
  • Thanks I will try that but I've got to ask what the.. is wrong with my code sample ? imho it is both concise and complete. – Etienne Oct 17 '14 at 05:38
  • A complete code example is one that I or anyone else can copy into an empty project and then compile and run without adding _anything_ else. It's tricky to do in some cases, such as this one, because the example has to include or generate test data needed to reproduce the problem. But while your code example is nicely concise, it's not complete. – Peter Duniho Oct 17 '14 at 05:42
  • Rarely seen any of those in StackOverflow but fair enough I can totally understand that people willing to help won't spend any time in generating the input necessary to reproduce the problem with my code. – Etienne Oct 17 '14 at 05:46
  • I agree they are rare. There are a _lot_ of really poorly-asked questions on SO. And I agree the one you posted here is one of the better ones. But at the end of the day, given that there's not even a guarantee of success, few if any people will spend much time trying to flesh an existing example out so that it runs and reproduces the problem. So you're left with purely speculative answers, which are never as good as to-the-point, fully-informed ones. :) – Peter Duniho Oct 17 '14 at 05:50
  • Using ZipArchive didn't change, but I think my problem is that the IIS server running from Visual Studio is 32 bit and I'm reaching the natural memory limit. – Etienne Oct 17 '14 at 07:48
  • I guess maybe I should have clarified: regardless of which .zip implementation you go with, the other thing you'll want to do to ensure against memory limitations is to write directly to disk, instead of to a MemoryStream. It looks like you're doing the latter in your initial example, and if you are still doing that with the .NET ZipArchive class, then yes...memory space is still going to be an issue. If you write directly to the disk, then the only thing you need to worry about is running out of disk space. :) – Peter Duniho Oct 17 '14 at 07:57
0

I gave up trying to use MemoryStream even though being on a 64bit system with 16gb of memory I should have been safe on that side.

The relevant topic I found was: OutOfMemoryException while populating MemoryStream: 256MB allocation on 16GB system

And using a temporary file to write/read the data instead of memory.

Community
  • 1
  • 1
Etienne
  • 1,058
  • 11
  • 22