5

Context

I'm working on upgrading a .NET library to support 64-bit. That library performs various operation directly in memory of other processes on Windows. I have to choose between the two types IntPtr (max positive value 7FFF'FFFF'FFFF'FFFF) or UIntPtr (max positive value FFFF'FFFF'FFFF'FFFF) to handle my memory pointers. There is a lot of information on the Web about the two. IntPtr seems to be the de facto agreed choice, as it is CLS-compliant and most of .NET API rely on that (ref Marshal from InteropServices).

The question

I decided to open a 64-bit process and inspect the allocated memory regions, as well as the loaded modules in the process to see if it would be valuable to support unsigned pointers using UIntPtr (addresses > 7FFF'FFFF'FFFF'FFFF). As illustrated in the screenshot below, it seems the memory addresses does not load symbols, nor allocate memory over 7FFF'FFFF'FFFF. Is there a specific reason for doing so ? May Windows allocate memory regions over that value in some cases ?

Memory allocation on Cheat Engine (64-bit)

Community
  • 1
  • 1
Jämes
  • 6,945
  • 4
  • 40
  • 56

3 Answers3

5

In Windows each process has only an address space of 8TB, therefore the upper limit for user code is 0x7FF'FFFF'FFFF

The range of virtual addresses that is available to a process is called the virtual address space for the process. Each user-mode process has its own private virtual address space. For a 32-bit process, the virtual address space is usually the 2-gigabyte range 0x00000000 through 0x7FFFFFFF. For a 64-bit process, the virtual address space is the 8-terabyte range 0x000'00000000 through 0x7FF'FFFFFFFF. A range of virtual addresses is sometimes called a range of virtual memory.

This diagram illustrates some of the key features of virtual address spaces.

some of the key features of virtual address spaces

https://learn.microsoft.com/en-us/windows-hardware/drivers/gettingstarted/virtual-address-spaces

The upper 248TB belongs to kernel mode, summing up to 256TB of address space, which is addressed by 48 bits. That means the highest possible positive address is 247-1 = 0x7FFF'FFFF'FFFF

In 64-bit Windows, the theoretical amount of virtual address space is 2^64 bytes (16 exabytes), but only a small portion of the 16-exabyte range is actually used. The 8-terabyte range from 0x000'00000000 through 0x7FF'FFFFFFFF is used for user space, and portions of the 248-terabyte range from 0xFFFF0800'00000000 through 0xFFFFFFFF'FFFFFFFF are used for system space.

Update:

As commented below, in Windows 8.1 and Windows Server 2012 R2 or later the user/kernel address space split is 128/128TB which sums up to the same 256TB space

The significant part is 48-bit wide probably because most current x86-64 implementations use 48-bit virtual address

The original implementation of the AMD64 architecture implemented 40-bit physical addresses and so could address up to 1 TB (240 bytes) of RAM. Current implementations of the AMD64 architecture (starting from AMD 10h microarchitecture) extend this to 48-bit physical addresses and therefore can address up to 256 TB of RAM. The architecture permits extending this to 52 bits in the future (limited by the page table entry format); this would allow addressing of up to 4 PB of RAM.

https://en.wikipedia.org/wiki/X86-64#Architectural_features

Community
  • 1
  • 1
phuclv
  • 37,963
  • 15
  • 156
  • 475
  • 2
    Starting with Windows 8.1, the user-mode virtual address space is 128 TB (refer to [Memory Limits for Windows and Windows Server Releases](https://learn.microsoft.com/en-us/windows/desktop/memory/memory-limits-for-windows-releases), i.e. the full 47 bits that are currently available to x64 user-mode code. – Eryk Sun Mar 26 '19 at 15:17
2

Due to specifications on x64 you can depend on userspace pointers always fitting in IntPtr. You cannot depend on a smaller space. The CPU could get more address lines in the future. When this happened between Windows 8 and Windows 8.1, no backwards compatibility flag was added.

In fact you got both positive and negative pointers in x86, but storing pointers in IntPtr worked anyway because of the no-mans-land at 0x7FFF0000 and the null trap range at 0x00000000.

I don't think you're trying to do tagged pointers, but if you are, the only acceptable way to do tagged pointers is the bottom two bits.

Joshua
  • 40,822
  • 8
  • 72
  • 132
  • To clarify, the change in Windows 8.1 was removing internal limitations in order to make the full 47-bit range of x64 available to user-mode addresses. That's not directly related to the previous statement that "[t]he CPU could get more address lines". – Eryk Sun Mar 26 '19 at 15:25
  • Tagged pointers can use the top 7 bits for now, as long as they redo sign-extension before use. (Or more simply, use the top byte and zero-extend for user-space pointers.) Upcoming PML5 hardware will allow 57-bit virtual addresses, so the low half of that range is 56 bits, leaving the top 8 unused. But yes, if you only need a couple bits, the low bits are more efficient to work with. `and reg64, -4` can clear them cheaply with an 8-bit sign-extended immediate, instead of needing 2 shifts or a 64-bit constant. – Peter Cordes Jun 08 '20 at 05:00
  • @PeterCordes: What will happen to your code when a CPU comes out with all 64 address lines? – Joshua Jun 08 '20 at 15:23
  • @Joshua: You'll keep running an OS that "only" uses 5-level page tables. (5 levels * 9 bits per level, plus 12 page-offset bits, equal 57. [Why in 64bit the virtual address are 4 bits short (48bit long) compared with the physical address (52 bit long)?](https://stackoverflow.com/q/46509152) has a diagram. Also, that makes the interesting point that x86-64's current page-table format only supports mapping virtual to 52-bit physical addresses, and thus more than 57 virtual address bits probably not worth doing.) – Peter Cordes Jun 08 '20 at 15:31
  • Or in the far future with a whole new HW page-table format with full 64-bit addresses, you'll use an OS that has a per-process backwards-compat "personality" flag to only choose random addresses in the low 47 or 56 bits, kind of like current Linux `MAP_32BIT` mmap flag. And of course most systems won't have that much RAM, and presumably there will be a way to use fewer levels of page tables, or traditional x86-64 page table mode with only 48 or 57-bit virtual addressing, – Peter Cordes Jun 08 '20 at 15:32
1

In Windows, the maximum virtual memory address is 7FFF'FFFF'FFFF'FFFF, i.e. it is not possible to allocate memory beyond this address. Historically, the first 64-bit processors from AMD and Intel (as per AMD64 specs) supported only addresses of 48 bits. Hence the limit.

See more details here: http://www.alex-ionescu.com/?p=50 and https://blogs.technet.microsoft.com/markrussinovich/2008/11/17/pushing-the-limits-of-windows-virtual-memory/

Nick
  • 4,787
  • 2
  • 18
  • 24
  • 2
    The limit for x64 user-mode code is 7FFF'FFFF'FFFF. Bits 47-63 have to be either all 0 (user mode) or all 1 (kernel mode). Interesting as that is, `IntPtr` would be adequate even if we had the full 64-bit range split equally between user and kernel mode. – Eryk Sun Mar 26 '19 at 14:23