Array size based on available physical memory

Question

I am trying to make an encryption algorithm. I can read a file and convert it to bytes without any problems, and am saving the bytes in a byteArray.

The problem is I am currently creating the array size like this:

byte[] FileArray =new byte[10000000];
FileStream TheFileStream = new FileStream(FilePath.Text, FileMode.Open);
BinaryReader TheFileBinary = new BinaryReader(TheFileStream);

for (int i = 0; i < TheFileStream.Length; i++) {
    FileArray = TheFileBinary.ReadBytes(10000000);
    // I call a function here
    if (TheFileStream.Position == TheFileStream.Length)
        break;
}

However, I don't want the array size to be fixed, because if I make it 1000000 (as an example), other machines with small memory size might face a problem. I need to find the Idle size of a memory size for each machine, how can I set the array size dynamically based on the free unallocated memory space, to be used where I can put it in the byteArray?

I have noticed the larger the Arraysize the faster it reads, so I don't want to make it too small either. I would really appreciate the help.

`am trying to make an encryption algorithm` That's probably a mistake. Secure crypto is _hard_. — SLaks, Mar 12 '12 at 18:54
The ~10MB you're asking for is a pretty small amount. Why are you worried about memory issues here? Have you run into problems? Do you anticipate specific problems in the future on some device where 10MB is a lot? — dlev, Mar 12 '12 at 18:55
The (arguably) _better_ approach would be to make your own Stream. — Brad Christie, Mar 12 '12 at 18:59
This might be a dumb question, but why not use `List` if you want a dynamic-sized array? — qJake, Mar 12 '12 at 19:11
He doesn't want a dynamic-sized array he wants an array who size is decided at runtime. — ClassicThunder, Mar 12 '12 at 19:17

SPFiredrake · Answer 1 · 2012-03-12T21:32:49.863

If you're really worried about space, then use a simple List, read in chunks of the stream at a time (say 1024), and call the AddRange method on the list. After you're done, call ToArray on the List, and now you have a properly size byte array.

List<byte> byteArr = new List<byte>();
byte[] buffer = new byte[1024];
int bytesRead = 0;
using(FileStream TheFileStream = new FileStream(FilePath.Text, FileMode.Open))
{
    while((bytesRead = stream.Read(buffer, 0, buffer.Length)) > 0)
        byteArr.AddRange(buffer);
}
buffer = byteArr.ToArray();
// call your method here.

Edit: It's still preferable to read it in chunks for larger files. You can of course play with the buffer size however you want, but 1024 is usually a good starting point. Doing a read of the entire file will ultimately DOUBLE the memory, as you also have to deal with the internal read buffer being the size of the stream (on top of your own buffer). Breaking up the reads into chunks only takes FileStream.Length + <buffer size> memory as opposed to FileStream.Length * 2. Just something to keep in mind...

byte[] buffer = null;
using(FileStream TheFileStream = new FileStream(FilePath.Text, FileMode.Open))
{
    buffer = new byte[TheFileStream.Length];
    int offset = 0;
    while((bytesRead = stream.Read(buffer, offset, 1024)) > 0)
        offset += bytesRead;
    // Or just TheFileStream.Read(buffer, 0, buffer.Length) if it's small enough.
}

You're reading the file in chunks but ultimately end up with a copy of the entire file stored in both `byteArray` and `buffer`. The author is worried about memory usage, and only wants one chunk of the file to be in memory at a time. — Lance U. Matthews, Mar 12 '12 at 19:37

score 0 · Answer 2 · edited May 23 '17 at 11:48

0

The FileStream keeps track of how many bytes are in the file. Just use the Length property.

FileStream TheFileStream = new FileStream(FilePath.Text, FileMode.Open);
BinaryReader TheFileBinary = new BinaryReader(TheFileStream);
byte[] FileArray = TheFileBinary.ReadBytes(TheFileStream.Length);

Okay reread the question and finnaly found the part of it that was a question, "how can I know the free unallocated memory space so I can put it in the byteArray". Anyways I suggest you take a look at this question along with its highest rated comment.

edited May 23 '17 at 11:48

Community

1
1

answered Mar 12 '12 at 19:12

ClassicThunder

1,896
16
25

The author is trying to read and process the file in optimally-sized chunks, whereas this will read the entire file into an array all at once. Also, the `FileStream.Length` property is misspelled. – Lance U. Matthews Mar 12 '12 at 19:20

score 0 · Answer 3 · answered Mar 13 '12 at 02:27

You can use WMI to retrieve the instance of the Win32_OperatingSystem class and base your memory calculations off of the FreePhysicalMemory or TotalVisibleMemorySize properties:

static ulong GetAvailableMemoryKilobytes()
{
    const string memoryPropertyName = "FreePhysicalMemory";

    using (ManagementObject operatingSystem = new ManagementObject("Win32_OperatingSystem=@"))
        return (ulong) operatingSystem[memoryPropertyName];
}

static ulong GetTotalMemoryKilobytes()
{
    const string memoryPropertyName = "TotalVisibleMemorySize";

    using (ManagementObject operatingSystem = new ManagementObject("Win32_OperatingSystem=@"))
        return (ulong) operatingSystem[memoryPropertyName];
}

Then pass the result of either method to a method like this to scale the size of your read buffer to the memory of the local machine:

static int GetBufferSize(ulong memoryKilobytes)
{
    const int bufferStepSize = 256;       // 256 kilobytes of buffer...
    const int memoryStepSize = 128 * 1024;// ...for every 128 megabytes of memory...
    const int minBufferSize = 512;        // ...no less than 512 kilobytes...
    const int maxBufferSize = 10 * 1024;  // ...no more than 10 megabytes
    int bufferSize = bufferStepSize * ((int) memoryKilobytes / memoryStepSize);

    bufferSize = Math.Max(bufferSize, minBufferSize);
    bufferSize = Math.Min(bufferSize, maxBufferSize);

    return bufferSize;
}

Obviously, increasing your buffer size by 256 KB for every 128 MB of RAM seems a little silly, but these number are just examples of how you might scale your buffer size if you really wanted to do that. Unless you're reading many, many files at once, worrying about a buffer that's a few hundred kilobytes or a few megabytes might be more trouble than it's worth. You might be better off just benchmarking to see which sized buffer gives the best performance (it might not need to be as large as you think) and using that.

Now you can simply update your code like this:

ulong memoryKilobytes =
    GetAvailableMemoryKilobytes();
    // ...or GetTotalMemoryKilobytes();
int bufferSize = GetBufferSize(memoryKilobytes);

using (FileStream TheFileStream = new FileStream(FilePath.Text, FileMode.Open))
{
    byte[] FileArray = new byte[bufferSize];
    int readCount;

    while ((readCount = TheFileBinary.Read(FileArray, 0, bufferSize)) > 0)
    {
        // Call a method here, passing FileArray as a parameter
    }
}

Array size based on available physical memory

3 Answers3