0

I'm trying to save a large file (~2GB) using MemoryStream. My server has around 6-8GB of RAM, but even then when I load the file to a MemoryStream object, I get an OutOfMemoryException. I need the file loaded as a byte array to run an anti-virus process on it, so there's no option to chunk it.

Currently I'm loading the byte array this way:

    public static byte[] ReadBytes(this Stream inputStream)
    {
        if (inputStream == null)
        {
            return new byte[0];
        }

        if (inputStream.CanSeek && inputStream.Position > 0)
        {
            inputStream.Position = 0;
        }

        // Load by 16KB chunks
        var buffer = new byte[16 * 1024];

        using (var ms = new MemoryStream())
        {
            int read;
            while ((read = inputStream.Read(buffer, 0, buffer.Length)) > 0)
            {
                ms.Write(buffer, 0, read);
            }
            if (inputStream.CanSeek) inputStream.Position = 0;
            return ms.ToArray();
        }
    }

Is there something I can do to avoid this error?

Thank you.

Heathcliff
  • 3,048
  • 4
  • 25
  • 44
  • 1
    Specifying a size for the MemoryStream upfront would probably help a bit, but it's putting off the inevitable. The thing is that `MemoryStream` needs ~2GB of *contiguous* memory, which might not be available. If you *need* it to be in memory at once, and you *need* it to be in a `Stream`, you might be low on options. If you can allocate a ~2GB byte array of storage at startup and then re-use that thereafter, you should get a bit more reliability. – canton7 Jan 13 '20 at 17:34
  • 1
    Does this answer your question? [.NET Out Of Memory Exception - Used 1.3GB but have 16GB installed](https://stackoverflow.com/questions/14186256/net-out-of-memory-exception-used-1-3gb-but-have-16gb-installed) – Sani Huttunen Jan 13 '20 at 17:34
  • 3
    >> "I need the file loaded as a byte array to run an anti-virus process on it, so there's no option to chunk it." - I would elaborate on why you think this... I can't see any reason you would need the ENTIRE file to be loaded at once... If it does I'd just look for a better 'anti-virus process' – Milney Jan 13 '20 at 17:35
  • @Milney I'm using Symantec's antivirus scan with `StreamScanRequest`. It needs the stream. – Heathcliff Jan 13 '20 at 18:13
  • If it just needs a stream, can't you pass it a FileStream? – canton7 Jan 13 '20 at 18:35
  • @canton7 It actually needs a `byte[]` – Heathcliff Jan 13 '20 at 18:44
  • From a quick Google, there are `Send` and `Finish` methods you can use to stream data in? – canton7 Jan 13 '20 at 18:52

0 Answers0