23

I'm running the following method on my development IIS server (from VS2010 IDE) on a 64-bit Windows 7 machine with 16GB of installed RAM:

public static MemoryStream copyStreamIntoMemoryStream(Stream stream)
{
    long uiLen = stream.Length;
    byte[] buff = new byte[0x8000];

    int nSz;
    MemoryStream ms = new MemoryStream();
    try
    {
        while ((nSz = stream.Read(buff, 0, buff.Length)) != 0)
        {
            ms.Write(buff, 0, nSz);
        }
    }
    finally
    {
        Debug.WriteLine("Alloc size=" + ms.Length);
    }

    return ms;
}

and I get the System.OutOfMemoryException on this line:

ms.Write(buff, 0, nSz);

That is thrown when 268435456 bytes are allocated:

Alloc size=268435456

which is 0x10000000 or 256 MB. So I'm wondering if there's some global setting that I need to set to make it work?

Here's a screenshot of the configuration setting for the project: enter image description here

c00000fd
  • 20,994
  • 29
  • 177
  • 400
  • Can you try setting a larger initial size in the [`MemoryStream` constructor](http://msdn.microsoft.com/en-us/library/bx3c0489.aspx) if you know approximately how big it will be? – Jonathon Reinhart Mar 24 '13 at 04:11
  • @JonathonReinhart: Just tried it. The `stream.Length` is set to 0. It's a ZIP archive stream, so I guess it's not provided up front. – c00000fd Mar 24 '13 at 04:18
  • 2
    Well the problem is that MemoryStream doubles the size of its internal buffer every time its exceeded. So maybe you could try setting a 512MB or 1GB initial size. – Jonathon Reinhart Mar 24 '13 at 04:20
  • Jon, I tried setting larger sizes but found that simply moved the problem: instead of dying while expanding, it died up-front in the New. This is really frustrating, because in my project it happens at 10 MB because I'm feeding from one stream to another. – Maury Markowitz Apr 21 '14 at 13:12
  • Possible alternative for MemoryStream here http://www.codeproject.com/Articles/348590/A-replacement-for-MemoryStream and here https://memorytributary.codeplex.com/ (Haven't tried it myself.) – RenniePet Dec 27 '14 at 05:30

2 Answers2

34

Short answer - dev server is 32bit process.

Long answer for "why just 256Mb?"

First of all, let's understand how it works.

MemoryStream has internal byte[] buffer to keep all the data. It cannot predict exact size of this buffer, so it just initializes it with some initial value.

Position and Length properties don't reflect actual buffer size - they are logical values to reflect how many bytes is written, and easily may be smaller than actual physical buffer size.

When this internal buffer can not fit all the data, it should be "re-sized", but in real life it means creating new buffer twice as size as previous one, and then copying data from old buffer to new buffer.

So, if the length of your buffer is 256Mb, and you need new data to be written, this means that .Net need to find yet another 512Mb block of data - having all the rest in place, so heap should be at least 768Mb on the moment of memory allocation when you receive OutOfMemory.

Also please notice that by default no single object, including arrays, in .Net can take more than 2Gb in size.

Ok, so here is the sample piece which simulates what's happening:

        byte[] buf = new byte[32768 - 10];

        for (; ; )
        {
            long newSize = (long)buf.Length * 2;
            Console.WriteLine(newSize);

            if (newSize > int.MaxValue)
            {
                Console.WriteLine("Now we reach the max 2Gb per single object, stopping");
                break;
            }

            var newbuf = new byte[newSize];
            Array.Copy(buf, newbuf, buf.Length);
            buf = newbuf;
        }

If it built in x64/AnyCPU and runs from console - everything is ok.

If it built across x86 - it fails in console.

If you put it to say Page_Load, built in x64, and open from VS.Net web server - it fails.

If you do the same with IIS - everything is ok.

Hope this helps.

Lanorkin
  • 7,310
  • 2
  • 42
  • 60
  • 14
    Most importantly, the 512 MB being allocated must be a contiguous chunk of memory. It's not that there isn't enough address space available in 32-bit; it's just that the address space is fragmented. A Stream implementation that uses multiple smaller byte arrays could get much closer to the 2 GB limit. – Daniel Mar 24 '13 at 19:34
  • Thank you for the explanation. On a different subject though -- why can't OS defragment its internal RAM mapping, say, while computer is not in use? Because it's ridiculous that it can't allocate 512MB of data. What, is it Windows 95? Sometimes I wonder if Microsoft did anything special in Windows 7 since those old days... – c00000fd Mar 24 '13 at 23:53
  • It's not as bad as you may think ) Read this answer for example http://stackoverflow.com/a/5243503/2170171 In most cases 512Mb will be allocated without any issues on 16Gb RAM machine, here is such specific case http://stackoverflow.com/questions/686950/large-object-heap-fragmentation – Lanorkin Mar 25 '13 at 08:15
  • I know this is old, but I wonder if Stream.CopyTo() method does not fix the memory allocation problem for stream lengths less than int.MaxValue? In which case, the following code may be less prone to OutOfMemory exceptions: `if (int.MaxValue >= memoryStream.Length) { MemoryStream memoryStream = new MemoryStream(); stream.CopyTo(memoryStream); memoryStream.Capacity = (int) memoryStream.Length; // Optional but helps eliminate null padding. }` – Teorist Sep 28 '18 at 18:21
  • 1
    @Teorist default `CopyTo` implementation does exactly the same as the code in question - reads source in buffer and writes destination in chunks, thus leading to internal buffer reallocates. You can see that in `Stream` references sources here https://github.com/Microsoft/referencesource/blob/master/mscorlib/system/io/stream.cs#L218 To avoid buffer reallocation, we need to 1) be able to know source stream length in advance (not all streams allows that) and 2) preallocate whole chunk of memory in destination at once – Lanorkin Oct 02 '18 at 09:13
6

If you are using default VS development server you are running code in x86/32 bit process. If you are using full IIS - most likely in IIS particular AppPool is configured to runs in x86 (32 bit mode) and as result have very limited address space (2GB unless you have marked your application as Large Address Aware).

In case of IIS make sure you have configured app polls to run x64 (not sure what is default). Make sure your code is target is set to AnyCPU or x64.

For standalone C# applications - by default they are compiled with x86 or AnyCPU/Prefer x86 - change target platform to x64.

To get x64 support for IIS you can either install full IIS or install IIS Express 8.0 ( 7.5 that comes with Windows 7 is 32 bit only) from Download IIS 8.0 Express.

Side notes:

Community
  • 1
  • 1
Alexei Levenkov
  • 98,904
  • 14
  • 127
  • 179
  • I just checked (see the screenshot above) and it seems like it's compiled as "Any CPU". Does it mean x64? – c00000fd Mar 24 '13 at 04:17
  • @c00000fd, ok. `MemoryStream` should be failing slightly later (around growing from 1GB)... so it is strange that you are hitting the error in 64 bit process that early... – Alexei Levenkov Mar 24 '13 at 04:20
  • Yeah. That's why I posted it here. I can check in the Task Manager to be sure that it runs as x64. What process does it run as in asp.net development server? – c00000fd Mar 24 '13 at 04:22
  • I think it's `WebDev.WebServer40.EXE`, isn't it? If so, it does run as a 32-bit process. – c00000fd Mar 24 '13 at 04:25
  • Oh, wow. Thanks. It's pretty lame that they haven't written a 64-bit version. Can I install IIS on Windows 7 that is not a Server version? – c00000fd Mar 24 '13 at 04:38
  • Thanks. Just installed it on my Windows 7 Ultimate. After changing the project properties to run in a local IIS, it now runs thru `iisexpress.exe` that is also a 32bit process. What am I doing wrong? – c00000fd Mar 24 '13 at 04:55
  • @c00000fd, I've updated answer with steps to get IIS Express 8 (with x64 support), but you probably did not configure your solution to use freshly installed IIS - try going to http://localhost as per instructions and see if you have w3wp process running (in Task manager make sure to click "show from all users"). – Alexei Levenkov Mar 24 '13 at 19:39