0

System information: Linux version 2.6.32-573.12.1.el6.x86_64 (mockbuild@x86-031.build.eng.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) ) #1 SMP Mon Nov 23 12:55:32 EST 2015

RAM 48 GB

Problem: I want to malloc() 100 GB memory. But it fail to allocate on redhat system.

I find that 100GB can be allocated in macOS with 8 GB RAM (clang compile). I am very confuse about that.

Maybe lazy allocation described in this link? Why malloc() doesn't stop on OS X?

But why linux system can not? I try ubuntu and redhat, both fail to to that.

Result: After investigation, I find that the following two steps will let malloc() unlimited:

  1. echo "1" > /proc/sys/vm/overcommit_memory

  2. ulimit -v unlimited

Community
  • 1
  • 1
Yuansheng liu
  • 165
  • 1
  • 2
  • 10

1 Answers1

0

Possible reasons:

  1. The system is out of physical RAM or swap space.

  2. In 32 bit mode, the process size limit was hit.

Possible solutions:

  1. Reduce memory load on the system.

  2. Increase physical memory or swap space.

  3. Check if swap backing store is full.

I think these point can help you to understand malloc failure issue.

Input validation. For example, you've asked for many gigabytes of memory in a single allocation. The exact limit (if any) differs by malloc implementation; POSIX says nothing about a maximum value, short of its type being a size_t.

Allocation failures. With a data segment-based malloc, this means brk failed. This occurs when the new data segment is invalid, the system is low on memory, or the process has exceeded its maximum data segment size (as specified by RLIMIT_DATA). With a mapping-based malloc, this means mmap failed. This occurs when the process is out of virtual memory (specified by RLIMIT_AS), has exceeded its limit on mappings, or has requested too large a mapping. Note most modern Unix systems use a combination of brk and mmap to implement malloc, choosing between the two based on the size of the requested allocation.

Sumit Gemini
  • 1,836
  • 1
  • 15
  • 19
  • This is a fact that the memory what I want to allocated more than RAM (100GB > 48GB). But the mac OS only 8GB RAM also can malloc(100GB). Why? – Yuansheng liu Nov 30 '16 at 13:09
  • The redhat system is 64bit release. Maybe not in 32 bit mode? Increasing physical memory is out of my intention. Swap size is 4GB. I will check the third suggestion. – Yuansheng liu Nov 30 '16 at 13:12
  • Have you checked swap space of Mac OS. Actually I faced this issue in java apps. At that time my machine was over loaded by processes and swap space was very less. – Sumit Gemini Nov 30 '16 at 13:13
  • @Yuanshengliu MacOS is a 64 bits system and has dynamic virtual memory, and it lets you allocate huge chunks of memory (its lazy allocation greatly helps in that case). Linux (depending on the version) has different limitations. – Jean-Baptiste Yunès Nov 30 '16 at 13:28
  • Actually I am not aware about Mac OS. So can not say more on it. – Sumit Gemini Nov 30 '16 at 13:34