How to find the maximum RAM that would be available to my process?

Question

I am trying to write a program to perform external merge sort on a massive dataset. As a first step, I need to split the dataset into chunks that would fit into RAM. I have the following questions:

Suppose my machine has x amount of RAM installed, is there a theoretical maximum limit on how much of it could be made available to my process?

When I run the below program, I get a non-zero value as available memory when it fails. Why does the memory allocation fail when there is still unused RAM left? there is still 2.8GB free RAM when the memory allocation fails. What explains the observed behavior?

        List<string> list = new List<string>();
        try
        {
            while (true)
            {
                list.Add("random string");
            }
        }
        catch(Exception e)
        {
            Microsoft.VisualBasic.Devices.ComputerInfo CI = new ComputerInfo();
            Console.WriteLine(CI.AvailablePhysicalMemory);
        }

If there are other programs running concurrently, how do I determine, how much RAM is available for use by the current process?

Possible duplicate of [How to get memory available or used in C#](http://stackoverflow.com/questions/750574/how-to-get-memory-available-or-used-in-c-sharp) — sujith karivelil, Dec 21 '15 at 05:29
"I am trying to write a program to perform external merge sort on a massive dataset." - place data in a database table and query in sort order? — Mitch Wheat, Dec 21 '15 at 05:33
@Mitch While that would work, here I am more interested in the external merge sort implementation from an academic standpoint — Aadith Ramia, Dec 21 '15 at 05:49
Perhaps you could do it experimentally? eg. http://stackoverflow.com/questions/13835778/how-does-catching-an-outofmemoryexception-work — Matt Searles, Dec 21 '15 at 06:00
Please see the updated answer below. I believe it provides exactly what you're looking for now. — Moon, Dec 21 '15 at 15:49
@Don Thanks Don...but I observed a strange behavior. I have edited my question with details..please see above — Aadith Ramia, Dec 23 '15 at 03:52
Firstly, make sure your building in 64-bit mode and not the default. Secondly, you will reach the maximum supported Array dimensions very quickly and error for that and NOT OutOfMemory. I will update my answer below to show you an example that FILLS up the RAM and then starts paging. — Moon, Dec 23 '15 at 04:19
I changed the build target to x64 and still get the same behavior. could you pls explain the significance of this change? and where did array dimensions come from? I am using a lsit...and I did hit the OutOfMemoryException — Aadith Ramia, Dec 23 '15 at 04:31
64-bit code has a larger address space than the default x86 and can allocate more RAM for all it's components. Did you check it was an OutOfMemory exception? Mine showed an exception of "Array dimensions exceeded supported range" There is a chance that you have not much RAM to start with and you may have Virtual Memory turned off. In that case it should use up all your remaining physical RAM per your original request. Also as you approach the limit the .NET GC will see the RAM pressure reaching it's limits and take actions to free up as much space as possible. — Moon, Dec 23 '15 at 04:52
One class that might interest you [`MemoryFailPoint`](https://msdn.microsoft.com/en-us/library/system.runtime.memoryfailpoint(v=vs.110).aspx), it lets you check if a chunk of memory would have thrown a OutOfMemoryException if you tried to do the same operation live. — Scott Chamberlain, Dec 23 '15 at 06:36
That is a fine class as long as it is understood that it will continue to succeed well after all physical RAM is used and will only fail once the system runs out of VIRTUAL memory. Except in cases where built in x86, etc.. where there is a 2GB limit. In those cases it would be more useful. — Moon, Dec 23 '15 at 18:08

Moon · Answer 1 · 2015-12-23T05:28:24.810

Here is what you're looking for: ComputerInfo.AvailablePhysicalMemory

Gets the total amount of free physical memory for the computer.

private ulong GetMaxAvailableRAM()
{
    Microsoft.VisualBasic.Devices.ComputerInfo CI = new ComputerInfo();
    return CI.AvailablePhysicalMemory;            
}

NOTE: You will need a to add a reference to Microsoft.VisualBasic

UPDATE: Your sample to fill the RAM will run into a few other limits first.
It will first hit OutOfMemory if your not building in 64-bit. You should change your solution to build for x64 = 64-bit within the Solution configuration:

Secondly your List has a maximum supported array dimension. By adding many small objects you will hit that limit first. Here is a quick and dirty example making a List of Lists of strings.
(This could have smaller code using Images etc... But I was trying to stay similar to your example.)

When this is run it will consume all of your RAM and eventually start paging to disk. Remember Windows has Virtual RAM which will eventually get used up, but it's much slower than regular RAM. Also, if it uses all that up, then it might not even be able to allocate the space to instantiate the ComputerInfo Class.

NOTE: Be careful, this code will consume all RAM and potentially make your system unstable.

List<List<string>> list = new List<List<string>>();
try
{
    for (UInt32 I = 0; I < 134217727; I++)
    {
        List<string> SubList = new List<string>();

        list.Add(SubList);

        for (UInt32 x = 0; x < 134217727; x++)
        {
            SubList.Add("random string");
        }
    }
}
catch (Exception Ex)
{
    Console.WriteLine(Ex.Message);
    Microsoft.VisualBasic.Devices.ComputerInfo CI = new ComputerInfo();
    Console.WriteLine(CI.AvailablePhysicalMemory);
}

NOTE: To prevent using the Disk you could try to use something like System.Security.SecureString which prevents itself from being written to disk, but it would be very very slow to accumulate enough to fill your RAM.

Here is a test run showing the Physical Memory usage. I started running at (1)

I suggest for your final implementation that you use the ComputerInfo.AvailablePhysicalMemory value to determine how much of your data you can load before loading it (leaving some for the OS). And also look to lock objects in memory (usually used for Marshaling, etc..) to prevent accidental use of Virtual Memory.

but the total amount of RAM won't be limiting factor in .NET (with certain exceptions) object size limit is 2GB — Mitch Wheat, Dec 21 '15 at 05:52
That is true for any single item, the code will need to allocate the RAM via other methods, like loading it in pieces to a collection, etc.. But the question was what the MAXIMUM RAM that would be available. And he said he's using a DataSet. Also there are more limitations if it's a 32bit OS. — Moon, Dec 21 '15 at 05:57
I built a C# app a while back that specifically tested how much RAM I Could allocate and ran it on a server with 96GB available. I was able to successfully allocate almost all the RAM, but it had to be allocated in pieces... adding to a collection. I suppose instead of calculating the total used RAM in the system, you could write a test program that does this, but then it would try to use VirtualRam too...so that would likely not be the best way to find how much RAM could be used without paging. — Moon, Dec 21 '15 at 06:14
I've edited the answer, there is a property in the ComputerInfo class which provides exactly what @Aadith was looking for. — Moon, Dec 21 '15 at 15:49

How to find the maximum RAM that would be available to my process?

1 Answers1