malloc function limit test

Question

I just learned about heaps and stack memory allocations and was curious how much i can allocate on heap. as i see heap allocation gives pointer to memory where requested amount of contiguous bytes are free. so i wrote following code to do a binary search for the the maximum amount i can allocate.

#include <iostream>
#include <chrono>
#include <cassert>
using namespace std;

int main ( void ) {

  size_t lo = 0, hi = -1ll; hi >>= 1; // i am doing this to avoid overflow.
  while (lo + 1 != hi) {
    char * ptr = NULL;
    ptr = (char*) malloc((lo+hi)>>1);
    if (ptr != NULL) lo = (lo+hi)>>1;
    else hi = (lo+hi)>>1;
    free(ptr);
  }

  cout << lo << " is amount of bytes of memory that we could allocate on this system's heap.\n";
  cout << (lo>>10) << " KB\n";
  cout << (lo>>20) << " MB\n";
  cout << (lo>>30) << " GB\n";

  // additional code.
  // this part is to read and write at n uniformly distant cells in 20GB of memory.

  auto start = chrono::high_resolution_clock::now();
  int * arr = (int *) malloc(5ll<<32); // 20GB
  bool ok = true;
  const int n = 10; // n means 5 * 2 ^ n cells to write and read.
  for (long long i=0; i<(5<<n); ++i) {
    assert((5ll<<30) > (i<<(30-n)));
    arr[i << (30-n)] = 23;
    ok &= arr[i << (30-n)] == 23;
  }
  auto end = chrono::high_resolution_clock::now();
  long long elapsed = chrono::duration_cast<chrono::milliseconds>(end - start).count();

  cout << "Elapsed time : " << elapsed << "ms\n";
  cout << (ok ? "ok is True\n" : "ok is False\n");
  
  free(arr);

  return 0;
}

Output

27672117247 is amount of bytes of memory that we could allocate on this system's heap.
27023551 KB
26390 MB
25 GB
Elapsed time : 13ms
ok is True

i know output can change with time on same machine. but i didn't expected it to be around ~22GB always, when my system's RAM has capacity of only 8GB. what is going on here ?

I assume this is linux? A Windows machine probably would have topped out around 16GB — Mooing Duck, Jul 13 '21 at 16:18
Related: [Understanding Virtual Address, Virtual Memory and Paging](https://stackoverflow.com/questions/22290347/understanding-virtual-address-virtual-memory-and-paging) — Mooing Duck, Jul 13 '21 at 16:19
it is on windows using mingw. thanks for that related text. ill give it a read. — Arshdeep Singh, Jul 13 '21 at 16:19
You'll also see fun stuff like declaring an array of immense size and barely see the needle move in memory consumed because you haven't USED much of that array. If you write into the beginning and end of the array you may find that you are only given a page at the beginning and a page at the end and absolutely no memory actually assigned to the array between those pages. — user4581301, Jul 13 '21 at 17:22
@user4581301 please elaborate that absolute no memory in between first and last page. what does that mean?. i tried writing and reading memory cells at uniformly distant n cells. 1k cells ran in like 15 ms (2^15) ran in like 250ms and (2^20) hanged my system. i do understand the concept of pages like they are called in RAM when required, but why did it hanged on 2^20 cell calls. i updated the code to show you new addition. — Arshdeep Singh, Jul 14 '21 at 05:19

score 2 · Answer 1 · answered Jul 13 '21 at 18:39

i didn't expected it to be around ~22GB always, when my system's RAM has capacity of only 8GB. what is going on here ?

Most modern OSes use Virtual Memory, which allows the OS to obtain memory from multiple sources, while hiding those details from apps. In this case, when your 8GB RAM is exhausted, the OS is likely using free space on the hard drive instead.

There is also the notion that Virtual Memory can be reserved and physically allocated as separate tasks. This allows allocated memory to not waste physical storage space until it is actually needed. Your code is not making use of the memory it is allocating. You may be reserving 22GB but not actually committing storage for 22GB.

malloc function limit test

1 Answers1