Fail to allocate a large amount of virtual memory

Question

I read that when you try to allocate more bytes than are available in RAM using malloc(), it allocates virtual memory. At least on Linux.

I want to allocate a huge amount of virtual memory, lets say 100 GB. So, I wrote something like this:

void* virtual_memory = malloc(100 gb int);

But the returned pointer is NULL.

I execute this code on a 64-bit Ubuntu virtual machine.

What am I doing wrong?

EDIT What I'm trying to achieve is to make htop tool displaying 100GB in VIRT column for my process.

UPDATE I CAN call malloc to allocate 2 GB at once 50 times

how big is your swap area? Maybe related: https://stackoverflow.com/a/11198816/7926064 — BNT, Dec 12 '17 at 07:41
And do your system *have* over a hundred gigabytes of virtual memory? What does [the `free` command](http://man7.org/linux/man-pages/man1/free.1.html) report? Perhaps you are misunderstanding how virtual memory works? — Some programmer dude, Dec 12 '17 at 07:41
Apart from above comments, what is `gb` ? I can imagine ways of breaking your code by bad definitions of that. Actually it seems hard to come up with a definition which makes your code compileable, with the following `int`. — Yunnosch, Dec 12 '17 at 07:42
@Yunnosch it's a placceholder for a real int64_t value used in my program. — Lingviston, Dec 12 '17 at 07:43
@Someprogrammerdude Idk actually. Virtual machine has 4GB of RAM, but I though that I can allocate any (or at least a much bigger than RAM) amount of virtual memory. — Lingviston, Dec 12 '17 at 07:45
you can't you should manage memory manually, like page file or using a file as a swap — nullqube, Dec 12 '17 at 07:45
Note that the memory returned by malloc is *contiguous* memory. So, not only do you need 100 GB to be available at all, but you need 100 GB to be available contiguously, and if that is not possible on the system then malloc will fail. — Remy Lebeau, Dec 12 '17 at 07:45
What do you intend to do if it **does** work? Anything but processing the array linearily seems doomed due to swapping time. — Yunnosch, Dec 12 '17 at 07:47
@Yunnosch I don't need to use this memory. I just want the htop tool to show that my process uses/has/whatever 100 gb of virtual memory. — Lingviston, Dec 12 '17 at 07:48
@Lingviston do you have any idea where virtual memory resides when it cannot be kept in physical memory? Think about that for a moment. Then reconsider the consequences of what you tried to do. Now think about why the OS might not be willing to allow you to grab 20 times more virtual memory than you have physical memory. — Disillusioned, Dec 12 '17 at 07:49
With respect to the OP, I have a feeling they are aware of the fact that the amount they are asking is in excess of the physical memory available. — Bathsheba, Dec 12 '17 at 07:51
@RemyLebeau Not quite. You need contiguous address space. The address spaces need not be mapped to contiguous memory. In fact the pages of address space can be in different locations. I.e. some in RAM, and some on disk. — Disillusioned, Dec 12 '17 at 07:52
@CraigYoung I personally have a feel that I can't allocate more than I have in all physical sources, but I have a strange task to show up 100gb of virtual memory for the process in htop tool. And it's claimed to be achievable via single line of code. — Lingviston, Dec 12 '17 at 07:56
"I read that when you try to allocate more bytes than are available in RAM using malloc(), it allocates virtual memory.", where did you read this non sense ? — Stargateur, Dec 12 '17 at 08:01
@Stargateur one of examples is here https://stackoverflow.com/questions/7504139/malloc-allocates-memory-more-than-ram — Lingviston, Dec 12 '17 at 08:14
@Lingviston I don't see any close quote in your link, from what you claim in your question, nobody tell that malloc allocate virtual memory when you allocate more bytes than are available in RAM. You have clearly misunderstood modern memory management. Plus C standard, doesn't care if memory is virtual or physical. malloc implementation are free to do what they want, return physical or virtual memory. — Stargateur, Dec 12 '17 at 09:12
@Stargateur OK, It ALWAYS allocates virtual memory. That doesn't change anything in this particular case. The question is about allocating more than you have, even though it might sound stupid. — Lingviston, Dec 12 '17 at 09:16
@Lingviston **Virtual don't mean unreal**. OS can't allocate memory that doesn't exist. Virtual memory is useful because there are many source of memory in a machine so the OS give you a virtual address so it can manage it easily. — Stargateur, Dec 12 '17 at 09:20
@Stargateur Please, read the comments above. They should give you a clear meaning about what needs to be achieved. I'm sorry if the original question was not clear for you, but this discussion doesn't really help or gives an answer to the question in any way. — Lingviston, Dec 12 '17 at 09:23
@chux sizeof(size_t) returns 8. While the type is recognized as long unsigned int. — Lingviston, Dec 12 '17 at 11:25
The `void* virtual_memory = malloc(100 gb int);` is not valid code - it is pseudo code as so one can get pseudo answers. Post true code that in itself shows "returned pointer is NULL." to get good answers. — chux - Reinstate Monica, Dec 12 '17 at 11:30
Note that allocating memory is not the same as using memory. See https://stackoverflow.com/q/19991623/2410359 — chux - Reinstate Monica, Dec 12 '17 at 11:32

Ajay Brahmakshatriya · Accepted Answer · 2017-12-12T08:26:25.037

2

I read that when you try to allocate more bytes than are available in RAM using malloc(), it allocates virtual memory

To start with, this is not correct. You always allocate virtual memory. This virtual memory is mapped to some area on the Physical memory(RAM) or the swap space. If the swap space + physical memory is less than 100 GBs, your allocation will fail. Also, the libc implementation might fail to allocate such a large amount, if it has some (programmable) limit set.

but I have a strange task to show up 100gb of virtual memory for the process in htop tool. And it's claimed to be achievable via single line of code.

Yes if you just need this much virtual memory, you can reserve memory but not commit it. You can read upon how mmap(*NIX) or VirtualAlloc(Windows) can be used for the same.

When you reserve a particular Virtual Address range, you tell the operating system that you intend to use this range, so other code can't use that. But it doesn't mean you can actually use it. This also means that it doesn't need a RAM/Swap backing. So you will be able to reserve arbitrarily large amount (less than 2^48 bytes on your 64 bit system of course).

Although I am not sure if htop will include that in the value it shows, you will have to try that out.

If this doesn't indeed add to your virtual memory count, you can map it to a file, instead of mapping it anonymously. This might create a 100 GB file on your system (assuming you have that much space), but you should even be able to read/write to it.

Following code can be used on linux -

int fd = open("temp.txt", O_RDWR | O_CREAT);
void* addr = mmap(NULL, 100 * GBS, PROT_WRITE | PROT_READ, MAP_PRIVATE, fd, 0);

edited Dec 12 '17 at 08:26

answered Dec 12 '17 at 08:14

Ajay Brahmakshatriya

8,993
3
26
49

Yes, it always allocates virtual memory. My wrong. Reserving memory without commit looks like exactly what I need. I'll try it out. Thanks! – Lingviston Dec 12 '17 at 08:17
Linux always reserves memory without commit. When you call `malloc` in Linux, no actual memory gets allocated until you try to use it. If it's failing, it is because you have it some hard per-process or per-system limit. – JeremyP Dec 12 '17 at 10:00
@JeremyP How can I remove those limits? I have a clean install of Ubuntu 17.04 64 bit on a virtual machine. – Lingviston Dec 12 '17 at 11:00
@Ajay even though I think your answer is correct unfortunately the memory allocated this way is not counted as virtual memory by the htop tool. – Lingviston Dec 12 '17 at 11:16
@Lingviston, did you also try with the file mapping way? I am sure that should show up in virtual memory. – Ajay Brahmakshatriya Dec 12 '17 at 11:45
@AjayBrahmakshatriya I tried with anonymous mmap. Will try with file either. – Lingviston Dec 12 '17 at 12:31
1

The following mmap works: mmap(0, amount, PROT_NONE, MAP_PRIVATE | MAP_ANONYMOUS | MAP_NORESERVE, -1, 0); – Lingviston Dec 12 '17 at 20:26
Just curious, you said: "So you will be able to reserve arbitrarily large amount (less than 2^48 bytes on your 64 bit system of course)." Are you implying there is memory limit of 2 ^48? If yes why? – Santosh Kale Jun 09 '21 at 08:11
1

@SantoshKale yes, there is a limit of 2^48 bytes on most x86-64 architectures. This is because only 48 bits of the 64 bits of the address lines (both virtual and physical are actually used). Rest of the 16 bits should all either be 0s or 1s (based on one of the used bit). This is a limit enforced in the hardware because hardware developers realized that we wouldn't need the entire address range in the near future. Using any other address that doesn't fit this constraint would result in a fault in the processor which would in most cases, send a SIGSEGV to your process. – Ajay Brahmakshatriya Jun 09 '21 at 13:39
1

@SantoshKale you can read about it here - https://stackoverflow.com/questions/6716946/why-do-x86-64-systems-have-only-a-48-bit-virtual-address-space – Ajay Brahmakshatriya Jun 09 '21 at 13:41

score 0 · Answer 2 · answered Dec 12 '17 at 11:27

The following code done the thing for me:

for (int i = 0; i < 50; ++i) {
    malloc(int_pow(2, 31));
}

Where int_pow is just a custom pow implementation, which operates integers. After running this app htop tool shows that it uses exactly 100GB of virtual memory.

Fail to allocate a large amount of virtual memory

2 Answers2