Kernel Mode memory size for an x86 LARGEADDRESSAWARE program on an x64 machine?

Question

The title pretty much sums it up. I have an application compiled for x86 platforms, which has the /largeaddressaware flag set. Running it on an x64 system, I get the expanded 4GB User Mode virtual memory "for free", without having to specify the /3GB boot option. On an x86 system this would mean that the Kernel Mode memory would only be 1GB, but since x64 systems can address more memory, does the Kernel Mode retain the 2GB or even get bumped up to 3GB?

Edit: To be clear, I'm wondering about per-process limits. The question stems from reading this article.

Edit 2: This question is not a duplicate of How much memory can a 32 bit process access on a 64 bit operating system? because that question addresses only the application-accessible memory, not the system accessible memory. If I have been misunderstanding and there is no memory reserved for the system for each process, I'd appreciate it if someone could write that up as an answer. I'm sure I'm not the first to be confused about this.

Answer 1

There's a few misconceptions that are confusing you.

First, let's look at 32-bit Windows. The virtual address space for each process has a certain part allocated to the process itself, and a certain part for whatever the kernel needs. However, all the processes share the same kernel memory - the fact that you even have kernel memory in your own virtual address space is basically a performance optimization to avoid having to switch address spaces when dealing with kernel objects and data in your application.

By default, this is a 1:1 split, so you get 2 GiB of user address space and 2 GiB of kernel address space. This was (ab)used by early 32-bit Windows software (when your computer might have had as little as 4 MiB of memory total with a 486 CPU or similar), because due to the way the memory was laid out, your user address space never had any pointers above the 2 GiB barriers - effectively giving you the highest bit of any pointer free for your own data. Often this was used to allow for a hybrid "if it fits, this is a value, otherwise it's a pointer to a structure" approach, saving memory and a bit of indirection. Since this is so wide-spread, the default has been the same split as in the early days to prevent compatibility issues. However, you also have a way to opt-in to a different split - 3 GiB of user space and 1 GiB of kernel space. This is what the /3GB option does. But that's not enough - your application must also opt-in using /LARGEADDRESSAWARE. This basically says "I don't do weird stuff with my pointers".

It should be noted that 32-bit OS or process doesn't necessarily mean you can only address 4 GiB of memory - it just limits what's directly accessible to the CPU at any point. For memory intensive server software, even the "32-bit" versions may have support for addressing a lot more memory - for example, the 32-bit MS SQL Server supports up to 64 GiB through AWE. This is basically another layer of virtualization which allows remapping the virtual address' physical addresses. In theory, there is no limit to the amount of memory you can address, with or without AWE - after all, nothing is preventing you from having your own hardware that acts as a memory mapped file, effectively giving you unlimited address space. Of course, like the days of segmented memory, it's not very easy to work with or practical :)

On 64-bit Windows, the /3GB no longer makes any sense and is ignored. The default address space split depends on the exact version of Windows, but is in the "terabytes and more" range, way out of the 32-bit limits. For modern Windows, this is usually 128 TiB user + 128 TiB kernel. 32-bit applications still have to use /LARGEADDRESSAWARE as before. However, since the kernel is now 64-bit, it can't be in the same address space as the user process anyway, so a 32-bit application on a 64-bit OS has full access to the 4 GiB of address space.

Of course, those limits are still well below what 64-bit is theoretically capable of addressing. However, most 64-bit CPUs actually can't address the whole 64-bit address space - the most common the last time I checked was just 48-bit. And surprise, surprise - that gives you 256 TiB of address space, the limit in Windows. Not a Microsoft conspiracy after all! :) This isn't something new, actually. The fact that Intel x86's 32-bit ALU is associated with a 32-bit address space is quite an outlier in CPU history - CPUs often have either higher and lower address space (for either virtual addressing or physical addressing) width than their ALU-width. The MS DOS typical limit of 1 MiB of addressable memory (with 640 kiB left over to user applications) comes from this as well - the "32-bit" CPUs of the time could only use 20-bit addresses.

Answer 2

As described in the article you link to, the 4GB address space available on a 32-bit CPU is divided into two parts: user mode or application address space, and kernel mode address space.

The user-mode address space is per-process. Each process has a different mapping between pages in the user-mode address space and physical or virtual memory.

The kernel-mode address space is the same regardless of which process is currently running. Otherwise, the address space would have to be remapped on every transition to kernel mode, which would be very inefficient. (The article does say that, but only very briefly: "the operating system makes its virtual memory visible in the address space of every process".)

By default, 32-bit Windows divides this up evenly, 2GB for user space and 2GB for kernel space, but it can be configured to divide it up 3GB/1GB instead.

On x64 Windows, the kernel runs in 64-bit mode, so it has access to the full address space permitted by the CPU, which is currently 48 bits or 256TB. The first x64 Windows releases used only 16TB of address space, divided evenly: 8TB for the application address space (for 64-bit applications) and 8TB for the kernel. In Windows 8.1 this was increased to use the entire 256TB allowed by the CPU, again divided evenly: 128TB for 64-bit applications, 128TB for the kernel.

32-bit applications run in the WOW64 emulation environment, with the CPU running in legacy mode. However, the kernel never runs in legacy mode. When a kernel transition is necessary, the CPU must be switched from legacy mode to long mode, which also means that it switches from the 32-bit address space to the 64-bit address space. x64 CPUs are designed so that this transition is efficient.

As a result, there is no need for any of the 32-bit address space to be reserved for the kernel.

To ensure backwards compatibility, a 32-bit process whose executable is not flagged as large address aware is still restricted to 2GB of address space. If the executable is large address aware, the process gets all 4GB.

You should note that this really is address space, not memory or even virtual memory. A 32-bit application can use file mappings and other methods to make use of more than 4GB of memory.

You should also note that the fact that the process has access to 2GB/3GB/4GB of address space does not mean that the application can use all of that space. Windows reserves some user-mode address space in each process for itself.

Address space and other limits are documented here: Memory Limits for Windows and Windows Server Releases.