-1

This code doesn't do anything practical I was just seeing what would happen.

As far as I can tell, the only two variables that are preserved are the (eventually) massive string, and a neglible size int tracking the string's length.

On my machine, the string gets to be about 0.75GB at which point the OutOfMemoryException occurs. At this stage Visual Studio is showing about 5GB of usage. So I'm wondering why there is a disparity.

var initialText = "Test content";
var text = initialText;
var length = text.Length;
while (true)
{
    try
    {
        var currentLength = text.Length;
        Console.WriteLine($"Current Length - {currentLength}");
        Console.WriteLine($"Current Size in GB - {System.Text.Encoding.UTF8.GetByteCount(text)/1024.0/1024.0/1024.0}");
        text = Convert.ToBase64String(System.Text.Encoding.UTF8.GetBytes(text));
        Console.WriteLine($"Change In Size - {currentLength / (length + 0.0)}");
        length = currentLength;
    }
    catch (OutOfMemoryException)
    {
        break;
    }
}

As a second question, when I begin to run the code my machine has about 11GB free according to Task Manager, and when it hits the exception, it's gone up by about 3GB, which doesn't tally up with the above numbers. Any ideas?

trincot
  • 317,000
  • 35
  • 244
  • 286
Geesh_SO
  • 2,156
  • 5
  • 31
  • 58
  • Ah interesting. I put `GC.Collect();` at the end of the loop and it had the same behaviour before as without it. Any idea how that ties into your comment? – Geesh_SO Oct 05 '19 at 23:34
  • 1
    You cannot reliably test the memory management having a debugger attached. You should run your app in Release mode without a debugger but with a memory profiling tool to get valid results. – dymanoid Oct 05 '19 at 23:41
  • I'm running the release version from the command line now and it's giving me identical behaviour as to running it in debug mode from Visual Studio. The extra commands I've added are `GC.Collect();` `GC.WaitForPendingFinalizers();` `GC.WaitForFullGCComplete();`. Thanks for the answers btw, it's been really interesting to read :) – Geesh_SO Oct 05 '19 at 23:47
  • There is a fundamental array size in .net, which is different for 32bit and 64bit applications. it doesn't matter who hits it, i.e your code or the code you call, if its hit you will get OutOfMemoryException. How much the operating system has given to .net, and how much .net has managed its not the issue here. – TheGeneral Oct 05 '19 at 23:49
  • In short, to see your issue, just create a bigger and bigger array (of any type). the rest of your code is just noise – TheGeneral Oct 05 '19 at 23:50
  • Ah I see. So in this case is the exception likely occurring due to the string (presumably an array of characters behind the scenes) simply going past its maximum size? – Geesh_SO Oct 05 '19 at 23:50
  • @Geesh_SO it would seem so, something somwhere is hitting the max allowed size for the allocation of an array (of some type) – TheGeneral Oct 05 '19 at 23:52
  • check out the remarks https://learn.microsoft.com/en-us/dotnet/api/system.array?redirectedfrom=MSDN&view=netframework-4.8 – TheGeneral Oct 05 '19 at 23:57

1 Answers1

1

First, strings in .net is a sequence of UTF-16 words, so each char takes 2 bytes. To get a size of the string in memory in bytes you need to multiply its length by 2 (ignoring CLR instance header).

Console.WriteLine($"Current Size in GB - {text.Length * 2.0 /1024/1024/1024}");

Another limitation is an array size in .NET, read remarks here as @TheGenral noted. There are 2 limits you can hit: max size(2GB) and max index.

Below is modified version of your test:

var text = "Test content";
long length = text.Length;
try
{

    while (true)
    {
        var currentLength = text.Length;
        Console.WriteLine($"Current Length - {currentLength}");
        Console.WriteLine($"Current Size in GB - {text.Length * 2.0 / 1024 / 1024 / 1024}");
        text += new string('a', 500 * 1024*1024);
        length = currentLength;
        GC.Collect();
    }
}
catch (OutOfMemoryException e)
{
    Console.WriteLine(e);
}

StringBuilder version difference:

var text = new StringBuilder("Test content");
...
text.Append('a', 500 * 1024*1024);

If you don't enable gcAllowVeryLargeObjects then you'll get OOM with 1B elements.

I was not able to get 2B elements using string concatenation, but if you rework this test using StringBuilder, then you can reach 2B of chars. In this case you'll hit a second limitation: arrays cannot hold more that 2 billion elements. Here is a discussion about upper limit.

In this thread max string length is discussed.

If you run this code in the Release mode you'll see process memory consumption almost equal to string size in console output.

Another interesting thing that I noticed and cannot explain is that with StringBuilder, gcAllowVeryLargeObjects and Debug mode I'm able to reach 4GB, but in the Release mode it nearly hits 3GB. Comments are welcome why that happens :)

fenixil
  • 2,106
  • 7
  • 13