0

How does memory allocation take place in this case? I observed that this is not the same as using a malloc on 1000000*10000 directly, which should have lead to 4*10GB (4 bytes per int) being allocated. However, this piece of code uses only 200MB on executing. 

for(int i=0;i<1000000;i++)
{
    int *A = (int*)malloc(sizeof(int)*10000);
}
Dhruv Mullick
  • 551
  • 9
  • 25
  • Do not cast the return value of malloc. How did you determine it doesn't allocate that much memory? Operating systems have methods of not actually giving physical memory out immediately necessarily. – Sami Kuhmonen Feb 07 '16 at 08:19
  • 2
    you are leaking memory except for the last iteration. – sunny1304 Feb 07 '16 at 08:19
  • Regardless: Read http://stackoverflow.com/questions/605845/do-i-cast-the-result-of-malloc – Idos Feb 07 '16 at 08:19
  • @SamiKuhmonen, I determined the memory allocated using Xcode IDE shows the amount of memory being used by the program. – Dhruv Mullick Feb 07 '16 at 08:24
  • 1
    Add a write to the allocated memory and see what happens. There are different numbers for memory "use" – Sami Kuhmonen Feb 07 '16 at 08:25

3 Answers3

2

As mentioned, there are differences in whether the memory is allocated in chunks or as one. The main reason why you're not seeing the memory being allocated is due to the operating systems lying about memory.

If you allocate a block of memory the allocation is virtual. Since all processes have lots of virtual memory available, it will usually succeed (unless you ask for insane amounts of memory, or the OS otherwise determines it's not going to work). The actual reservation of physical memory may occur after you actually use the memory.

So when you look at memory usage, there is not only one number but several. There is shared memory, there is memory that can't be paged out, there's the virtual allocated memory and then there's the actual memory in use.

If you change the code to actually use the memory, for example just write one byte to the allocated section, you will see completely different result. The OS has to handle the memory allocation and get the memory blocks in physical memory.

Also as mentioned you don't check that malloc succeeds. Maybe it succeeds for a few times and then doesn't allocate anything more.

This system also explains why sometimes a process might get killed due to low memory even though all allocations succeeded in all processes. The OS was just being too optimistic and thought it could give out more memory than actually was possible.

Sami Kuhmonen
  • 30,146
  • 9
  • 61
  • 74
  • I tried doing that, and suddenly I got a memory usage of 4GB, against the 200MB I was getting earlier. Can you please explain why the memory usage isn't 4*10^10 Bytes (40GB) ? – Dhruv Mullick Feb 07 '16 at 10:04
  • @DhruvMullick Did you check that all allocations succeeded? The OS also does paging and might not allocate full blocks if only a part is used. Not familiar how OS X does it under the hood exactly. – Sami Kuhmonen Feb 07 '16 at 10:40
  • When instead of just using A[0] = 5, I used A[0,1....1000] = 5, the memory increased dramatically. This makes sense with what you have just explained to me. – Dhruv Mullick Feb 07 '16 at 10:43
  • When I execute malloc 10^5 times, is it possible that the virtual memory block of `i th` iteration gets overwritten from the malloc in some other iteration? – Dhruv Mullick Feb 07 '16 at 10:43
  • @DhruvMullick The OS may page things out if it wants, so physical memory blocks may be reused, but virtual memory will all be unique. – Sami Kuhmonen Feb 07 '16 at 10:44
1

The difference is how the memory is allocated.

When you call 10K times malloc to allocate 10k of memory, 10G of virtual memory is allocated to your process. The resulting 10G of memory is not continuous. That is, you get 10k scattered blocks of memory whose size is 10K. Whereas, when you call malloc requesting 10G, malloc will try to allocate a continuos block of memory whose size is 10G.

According to the malloc manual page, malloc fails when it can't allocate the requested memory. You should check if the malloc is successful in your application, in order to understand if the memory has been correctly allocated.

Giuseppe Pes
  • 7,772
  • 3
  • 52
  • 90
  • I'm not sure if `malloc` always requires the allocated memory to be _continuos_ ?? – artm Feb 07 '16 at 08:36
  • When I execute malloc 10^5 times, is it possible that the memory block of `i th` iteration gets overwritten from the malloc in some other iteration? – Dhruv Mullick Feb 07 '16 at 10:36
  • @DhruvMullick No that's not possible. What can happen is that the i th call may fail if you don't have any memory left. – Giuseppe Pes Feb 07 '16 at 13:00
  • 1
    @artm the virtual memory will certainly continuous, but the real memory to which is mapped to may not be. In my answer I was referring to the virtual address space. – Giuseppe Pes Feb 07 '16 at 13:01
1
for(int i=0;i<1000000;i++)
{
    int *A = (int*)malloc(sizeof(int)*10000);
}

This is a perfect way of memory leak. You are allocating 1000000 times, each time sizeof(int)*10000 bytes. You're guaranteed to leak all of these allocated memory. As you declared A within the loop, so after the loop you do not have handle to that variable any more, and there's no way to free even the last chunk of memory you allocated.

And of course this is different from allocating 1000000*10000*sizeof(int) in one go. The former allocates 1000000 smaller chunks which are mostly to be scattered in many memory location. The latter tries to allocate one gigantic chunk, which likely to fails.

artm
  • 17,291
  • 6
  • 38
  • 54
  • But that does not explain why I am getting a lower memory usage report that what was expected. Won't `sizeof(int)*10000 bytes` be allocated 1000000 times? Then we should have a memory usage of 40GB – Dhruv Mullick Feb 07 '16 at 10:13