3

I am looking at the memory consumed by my app to make sure I am not allocating too much, and am confused as to what Gnome Resource Monitor is showing me. I have used the following pieces of code to allocate memory in two separate apps that are otherwise identical; they contain nothing other than this code and a scanf() call to pause execution whilst I grab the memory usage:

malloc(1024 * 1024 * 100);

and

char* p = new char[1204*1024*100];

The following image shows the memory usage of my app before and after each of these lines:

Memory use

Now, I have read a lot (but obviously not enough) about memory usage (including this SO question), and am having trouble differentiating between writeable memory and virtual memory. According to the linked question,

"Writeable memory is the amount of address space that your process has allocated with write privileges"

and

"Virtual memory is the address space that your application has allocated"

1) If I have allocated memory myself, surely it has write privileges?

2) The linked question also states (regarding malloc)

"...which won't actually allocate any memory. (See the rant at the end of the malloc(3) page for details.)"

I don't see any "rant", and my images show the virtual memory has increased! Can someone explain this please?

3) If I have purely the following code:

char* p = new char[100];

...the resource monitor shows that both Memory and Writeable Memory have increased by 8KB - the same as when I was allocating a full one megabyte! - with Virtual memory increasing by 0.1. What is happening here?

4) What column should I be looking at in the resource monitor to see how much memory my app is using?

Thanks very much in advance for participation, and sorry if have been unclear or missed anything that could have led me to find answers myself.

Community
  • 1
  • 1
Wad
  • 1,454
  • 1
  • 16
  • 33
  • 1
    I am not clear if there is an exact duplicate for your question, so I am not closing it. But the technical term that you are probably missing is "overcommit". A search on SO with `[C] overcommit` shows a lot of results. Please look a bit around and then come back and tell us if you still have more concrete questions. – Jens Gustedt Oct 16 '14 at 16:02
  • 1
    That and "lazy allocation" or "demand allocation"... When your application "allocates" more memory, Linux tends to just mark some additional virtual address space as in use, without actually allocating physical memory to populate that address space - the actual allocation happens later when you first attempt to access the new virtual addresses... – twalberg Oct 16 '14 at 16:10
  • 1
    OK. I have researched the recommended terms and found useful links. [This](http://www.etalabs.net/overcommit.html) talks about overcommit, and [this](http://www.win.tue.nl/~aeb/linux/lk/lk-9.html#ss9.6) has some example code. So I know now why my "Writeable" column doesn't increase until I touch my memory (via 'memset') but my "Virtual" one increases immediately. I'm still not clear why touching my memory changes it from "Virtual" to "writeable" however. The other question that still remains is why I have 8KB allocated for just 100 chars... – Wad Oct 16 '14 at 17:07
  • 1
    When you allocate a small block of memory, the memory manager will allocate a larger block and then split it into pieces to return to your code. You are probably just seeing that. – dohashi Oct 16 '14 at 17:19
  • 2
    Touching the memory does not "change it from virtual to writeable" - those are two completely different characteristics. All the memory mapped in a processes virtual address space is virtual. Some of it is writeable, some read only, some executable. Some is currently backed by real physical memory, some is not. Some may have been paged out to swap space, or be backed by files, etc... And it's all done in pagesize increments (hence your 8KB for 100 characters). – twalberg Oct 16 '14 at 17:20
  • @twalberg, your comment has come the closest to an actual answer. Whereas I, over the comments that have been posted, now have answers, I want the accepted "answer" to be useful for others in the future. Can I ask that you summarize your last comment as a proposed answer please? – Wad Oct 19 '14 at 18:00

2 Answers2

2

A more precise way to understand on Linux the memory usage of a running process is to use the proc(5) file system.

So, if your process pid is 1234, try

cat /proc/1234/maps

Notice that processes are having their address space in virtual memory. That address space can be changed by mmap(2) and other syscalls(2). For several efficency reasons malloc(3) and free avoid to make too much of these syscalls, and prefer to re-use previously free-d memory zones. So when your program is free-ing (or, in C++, delete-ing) some memory chunk, that chunk is often marked as re-usable but is not released back to the kernel (by e.g. munmap). Likewise, if you malloc only 100 bytes, your libc is allowed to e.g. request a whole megabyte using mmap (the next time you are calling malloc for e.g. 200 bytes, it will use part of that magabyte)

See also http://linuxatemyram.com/ and Advanced Linux Programming (and this question about memory overcommit)

Community
  • 1
  • 1
Basile Starynkevitch
  • 223,805
  • 18
  • 296
  • 547
  • Thanks for the comment Basile. However, whereas the information is useful, it doesn't really answer my question (and I mean from the point of view of someone looking at this question in the future). I want the accepted answer to be a reference to what I have asked. I'm sure you understand :) – Wad Oct 19 '14 at 17:59
1

The classes of memory reported by the Gnome resource monitor (and in fact, the vast majority of resource reporting tools) are not simply separate classes of memory - there is overlap between them because they are reporting on different characteristics of the memory. Some of those different characteristics include:

  • virtual vs physical - all memory in a processes address space on modern operating systems is virtual; that virtual address space is mapped to actual physical memory by the hardware capabilities of the CPU; how that mapping is done is a complex topic in itself, with a lot of differences between different architectures
  • memory access permissions - memory can be readable, writable, or executable, or any combination of the three (in theory - some combinations don't really make sense and so may actually not be allowed by hardware and/or software, but the point is that these permissions are treated separately)
  • resident vs non-resident - with a virtual memory system, much of the address space of a process may not actually be currently mapped to real physical memory, for a variety of reasons - it may not have been allocated yet; it may be part of the binary or one of the libraries, or even a data segment that has not yet been loaded because the program has not called for it yet; it may have been swapped out to a swap area to free up physical memory for a different program that needed it
  • shared vs private - parts of a processes virtual address space that are read-only (for example, the actual code of the program and most of the libraries) may be shared with other processes that use the same libraries or program - this is a big advantage for overall memory usage, as having 37 different xterm instances running does not mean that the code for xterm needs to be loaded 37 different times into memory - all the processes can share one copy of the code

Because of these, and a few other factors (IPC shared memory, memory-mapped files, physical devices that have memory regions mapped in hardware, etc.), determining the actual memory in use by any single process, or even the entire system, can be complicated.

twalberg
  • 59,951
  • 11
  • 89
  • 84