4

A server application I'm working on built using C++ on Windows runs out of memory when Virtual Size reaches somewhere around 2GB (32-bit application, with large address aware enabled). I notice, however, that Private Bytes is significantly smaller. The current statistics are:

Virtual Size: 2.6GB Private Bytes: 1.6GB

The difference in these two numbers is 1GB. So my questions are:

  1. What does this 1GB difference represent?
  2. Is my application running out of memory due to Virtual Size or Private Bytes?

I've also run my application through the VMMap utility and I notice that "Private Data" is usually an order of magnitude higher than the committed size. In other words, total size for Private Data might be 200MB, but the committed size is only 20MB. I'm not really sure what private data is, but based on my research so far it seems to indicate it's just part of the heap.

EDIT:

I've looked for memory leaks using Purify but I've not really found anything useful. Memory leaks in the form of memory without pointers doesn't seem to be the problem, but memory leaks in terms of memory being held onto for too long might be an issue, i haven't looked into that yet. However, the key is understanding why Virtual Size is causing the out of memory issues. Question #1 is the most important for me to understand this.

void.pointer
  • 24,859
  • 31
  • 132
  • 243
  • Perhaps you have one or more memory leaks that need to be fixed ? – Paul R Jul 02 '12 at 15:26
  • Committed = memory actually in use by application, private = memory reserved specifically for the application? And virtual size = amount of data paged on disk, I guess. I'd have to actually look into it to know for sure. – JAB Jul 02 '12 at 15:28
  • @JAB That's incorrect. I'll post an answer explaining. – Polynomial Jul 02 '12 at 15:28
  • If you're trying to use a User-Mode Dump Heap (UMDH) tool, the memory usage may be due to GFlag's collection of stack traces; see Paul Arnold's response to https://stackoverflow.com/questions/1831683/process-memory-increases-much-faster-with-gflags-ust – buzz3791 Mar 13 '18 at 18:50

4 Answers4

6

This is going to require a little explanation, so stick with me here.

First off, this topic is a confusing quagmire of conflicting terms, so please throw away all notions you have of "virtual memory" being anything to do with disks.

  • Physical memory is memory stored on a physical device. This usually refers to system RAM, but can also be disk buffers, NIC buffers, graphics card VRAM, etc.
  • Virtual memory is a set of physical address ranges mapped into user-mode (virtual) address ranges so that memory can be accessed in a safe and compartmentalised way.

A quick note on why we do this: if we gave processes direct memory addresses, we could only (feasibly) have a single memory store. This is inconvenient and bad for performance. When a virtual address translates to a physical address outside the range of the system memory (RAM), the processor issues a page fault. This signals an interrupt handler in the OS, which can then delegate the memory access operation to a different device. Useful!

On a 32-bit Windows system, the maximum amount of virtual memory that a process can address at any one point in time is 2GB. This can be increased to 3GB using AWE, or 4GB with /4GT and AWE. This does not mean that a process can only ever allocate 2GB (or 3GB / 4GB depending on the previously discussed settings) of memory. It just means that it cannot have concurrent access to more than that.

For example, if you open a memory-mapped file that is 1GB in size, your virtual memory usage will increase by 1GB. You're not touching the RAM, nor the disk, but you've allocated a block of virtual address space to your process. If you then want to allocate 1.1GB of RAM at the same time as having this memory mapped file available, you cannot. You must first unmap the file from your process. Remember that memory can still remain allocated and full of data, but not actually mapped into your process. If you have 8GB of RAM on your machine, you could fill 6GB of it up with data and map 2GB of it into your process. When you need to use a different section of that memory, you must unmap the existing block and map the other section.

So, onto the difference you're seeing:

  1. Private bytes tells you how many bytes of virtual device memory your process has mapped, excluding virtual memory shared with other processes (e.g. mapped files, global heap, etc).

  2. Working set tells you how many bytes of physical memory you are actively using. This includes physical memory, device buffers, and mapped files. It's a pretty strange figure, since it equates to touched physical memory + mapped virtual non-system memory. In general, you should completely ignore this figure. It's practically useless for debugging memory leaks.

  3. Virtual bytes is the total amount of virtual memory you have mapped.

The difference is that you've mapped shared virtual memory, for example a bunch of DLL files or a block of global heap, into your process. The difference indicates that the total size of these shared mappings is roughly 1GB.

Keep in mind that none of this has anything to do with swapping, which is the transfer of system memory pages to disk (the so-called "pagefile") to increase the availability of fast system resources (RAM). The naming of that file has caused no end of confusion in this area of Windows, and I'll be elated when Microsoft finally decide to call it "swap" rather than "virtual memory" or "page file".

Polynomial
  • 27,674
  • 12
  • 80
  • 107
  • Oh, and *definitely* give this a read: http://blogs.technet.com/b/markrussinovich/archive/2008/11/17/3155406.aspx Mark Russinovich is "da bawss" of this low-level memory stuff. – Polynomial Jul 02 '12 at 15:54
  • "This can be increased to 3GB using PAE". Actually PAE allows 32-bit OSes to access more than 4 GB of _physical_ memory. AWE is what allows 32-bit processes to access more than 2 GB (or 3 GB with 4GT) of user memory using memory windows. The 4GT is what allows applications to have up to 3 GB of mapped memory in the user part of the virtual address space. – Hristo Iliev Jul 02 '12 at 15:58
  • Sorry, got my terminology mixed up. Editing to fix. – Polynomial Jul 02 '12 at 15:59
  • I guess a follow-up question would be: why are the dlls introducing such high shared memory usage? – void.pointer Nov 07 '14 at 17:03
2
  1. Virtual Size vs Private Bytes explained: What is private bytes, virtual bytes, working set? (see quote below)
  2. Your application is likely to hit Virtual Size limit at 2 GB, esp. since you are seeing such behavior yourself
  3. /LARGEADDRESSAWARE extends the Virtual Size limit for your application in Win32 operating systems only when the system itself booted with /3GB AKA 4GT enabled

Virtual Bytes are the total virtual address space occupied by the entire process. This is like the working set, in the sense that it includes memory-mapped files (shared DLLs), but it also includes data in the standby list and data that has already been paged out and is sitting in a pagefile on disk somewhere. The total virtual bytes used by every process on a system under heavy load will add up to significantly more memory than the machine actually has.

So the relationships are:

  • Private Bytes are what your app has actually allocated, but include pagefile usage;
  • Working Set is the non-paged Private Bytes plus memory-mapped files;
  • Virtual Bytes are the Working Set plus paged Private Bytes and standby list.
Community
  • 1
  • 1
Roman R.
  • 68,205
  • 6
  • 94
  • 158
  • I have read the answer you linked more times than I can remember, but I'm still confused. I want to know specifically why the numbers are so different, but I can't even begin to identify that unless I know exactly what can cause the difference. So it's data on the standby list and/or pages sitting on disk? If that's the case, why doesn't Windows get rid of unused pages (thus reducing virtual size) instead of just failing? – void.pointer Jul 02 '12 at 15:37
  • A simple thing for starters: Private Bytes counts for `new` and `malloc` - as they succeed, there is a valid pointer for the allocated memory and hence they consume address space. However you have all the code also mapped into address space, not only your but all those API DLLs as well, they all have their addresses. And they are not Private Bytes. So if you open Process Explorer, or Visual Studio, Modules pane - you will see how many of the modules are loaded into process and they all count into the difference you asked about. – Roman R. Jul 02 '12 at 15:45
1

I had similar issue on my machine where the C/C++/.NET win32 application run out of memory. It consumed all 2GB of Virtual Addresses. It looked like Virtual Addresses Exhaustion because Process Explorer showed only about 900MB memory taken by the application. VMMap showed 1.6GB of Private Data and about 700MB uncommitted memory.

The reason was more than simple. The Application Verifier (C:\Windows\SysWOW64\appverif.exe) which was configured to test the application (Basics tests were marked - vfbasics.dll). After removing the application from Application Verifier it worked fine.

MikeV
  • 11
  • 2
  • I had the same problem. To keep using Application Verifier, I found that switching appverif.exe from _Full Page Heap-mode_ to just plain _Normal Page Heap-mode_ reverted the memory usage of the monitored process back to normal. Toggling between _Full-mode_ and _Normal-mode_ is done by toggling the _Full-checkbox_ in Application Verifier>Tests>Basics>Heap>Right-click menu>Properties. Or setting the windows registry _PageHeapFlags_ string value to 0x2 for _Normal-mode_. 0x3 is _Full-mode_. appverif.exe>Help>Heaps Stop Details>Heap details says _Normal_ force crash until _free()_. – buzz3791 Mar 13 '18 at 18:41
0

Just a comment: Enabling large address aware only signals the OS that it could safely divide the virtual address space of processes, created from that particular executable, in the 3:1 fashion instead of the usual 2:2. There are various reasons of why the application should explicitly identify that it supports 3:1 division but the most obvious one is that in 2:2 mode probing the MSB of a virtual address could be used to test if the address belongs to the kernel portion of the virtual address space or to the user portion. In 3:1 this test is no longer valid since the MSB is also 1 for 1/3rd of the user portion of the virtual address space. This requres that drivers which this application might call into should also be aware of possible 3:1 division and should use another method to test if a given virtual address belongs to userspace of not.

You still have to explicitly tell the kernel to enable support for 3:1 VA space division as Roman R. has pointed out.

Hristo Iliev
  • 72,659
  • 12
  • 135
  • 186
  • Just keep in mind that /3GB causes problems with the clipboard on large screen resolutions with not much RAM. Print-screen won't fit into the kernel's clipboard buffer, so the operation fails. Strangeness! – Polynomial Jul 02 '12 at 15:52