How to correctly allocate a large chunk of RAM with calloc/malloc in Windows 10?

Question

I need to allocate memory for a vector with n=10^9 (1 billion) rows using calloc or malloc but when I try to allocate this amount of memory the system crashes and returns me NULL, which I presumed to be the system not allowing me to allocate this big chunk of memory. I'm using Windows 10 in a 64-bit platform with 16 GB RAM. However, when I ran the same code in a Linux OS (Debian) the system actually allocated the amount I demanded, so now I'm wondering:

How can I allocate this big chunk using Windows 10, once I'm out of time to venture in Linux yet?

#include <stdlib.h>
#include <stdio.h>
#include <stdint.h>

int main(void) {

    uint32_t * a = calloc(1000000000, 4);
    printf("a = %08x\n", a);

    return 0;
}

Have a look at this question: https://stackoverflow.com/questions/181050/can-you-allocate-a-very-large-single-chunk-of-memory-4gb-in-c-or-c — daShier, Sep 11 '19 at 02:39
@daShier indeed what I thought. From Benoit's answer I can see windows does not let me allocate more than 4GB, is that the problem then? I mean, is it not possible to overcome this problem and allocate the memory needed in my computer other than using Linux? — Iron Maiden, Sep 11 '19 at 02:42
As an aside: why do you need such a huge vector? Perhaps you need to think about creating an index and buffer chunks of data? Another possibility (greatly depending on your data structure) is to think about using a database since they are really good at indexing and retrieving information in massive tables. — daShier, Sep 11 '19 at 02:53
@daShier I'm actually experimenting with it, thus going as deep as I can to learn how to manage memory allocation. Linux allowed to do what I wanted though. Anyway, I don't know how to create this index and later buff chunks of data once I'm a newbie in the CS field. So, can you help me with some tips or articles about it so I can study it? — Iron Maiden, Sep 11 '19 at 03:02
Worth reading [maximum memory which malloc can allocate](https://stackoverflow.com/questions/2798330/maximum-memory-which-malloc-can-allocate) — David C. Rankin, Sep 11 '19 at 05:48

SoronelHaetir · Answer 1 · 2019-09-11T04:06:28.087

The C run time won't let you do this but windows will, use the VirtualAlloc API instead of calloc. Specify NULL for the lpAddress parameter, MEM_COMMIT for flAllocationType and PAGE_READWRITE for flProtect. Also note that even though dwSize uses the "dw" decoration that is usually DWORD in this case the parameter is actually a SIZE_T which is 64-bit for 64-bit builds.

Code would look like:

#include <windows.h>

...

LPVOID pResult = VirtualAlloc(NULL, dwSize, MEM_COMMIT, PAGE_READWRITE);
if(NULL == pResult)
{ /* Handle allocation error */ }

Where dwSize is the number of bytes of memory you wish to allocate, you later use VirtualFree to release the allocated memory : VirtualFree(pResult, 0, MEM_RELEASE);

The dwSize parameter to VirtualFree is for use when you specify MEM_DECOMMIT (rather than MEM_RELEASE), allowing you to put memory back in the reserved but uncommitted state (meaning that actual pages have not yet been found to satisfy the allocation).

I'm a newbie in this field so I don't know exactly how to do that you've said. I've googled the VirtualAlloc API and found the function : LPVOID VirtualAlloc( LPVOID lpAddress, SIZE_T dwSize, DWORD flAllocationType, DWORD flProtect ); so I should replace it like this ? : LPVOID VirtualAlloc( LPVOID NULL, SIZE_T dwSize, DWORD MEM_COMMIT, DWORD PAGE_READWRITE ); — Iron Maiden, Sep 11 '19 at 03:08

How to correctly allocate a large chunk of RAM with calloc/malloc in Windows 10?

1 Answers1