36

In this document on p. 27 it says that text segment starts at 0x400000. Why was this particular address chosen? Is there any reason for that? The same address is chosen in GNU ld on Linux:

$ ld -verbose | grep -i text-segment
  PROVIDE (__executable_start = SEGMENT_START("text-segment", 0x400000)); . = SEGMENT_START("text-segment", 0x400000) + SIZEOF_HEADERS;

It's surprising because this address is bigger in 32-bit x86 executables:

$ ld -verbose | grep -i text-segment
  PROVIDE (__executable_start = SEGMENT_START("text-segment", 0x08048000)); . = SEGMENT_START("text-segment", 0x08048000) + SIZEOF_HEADERS;

I read this question which discusses why 0x080xxxxx address was chosen for i386 but it doesn't explain a change in x86_64. It's hard to find any explanation on that matter. Does anybody have a clue?

Peter Cordes
  • 328,167
  • 45
  • 605
  • 847
user1042840
  • 1,925
  • 2
  • 16
  • 32
  • https://www.uclibc.org/docs/psABI-x86_64.pdf is the latest version (0.99.7), [according to OSDev Wiki](http://wiki.osdev.org/System_V_ABI#Documents). – ivan_pozdeev Sep 25 '16 at 18:28
  • 0x400000 is 4MiB so this might have something to do with huge page support. Section 3.3.3 only allows page size up to 64KiB though. – ivan_pozdeev Sep 25 '16 at 18:46
  • @ivan_pozdeev: https://github.com/hjl-tools/x86-psABI/wiki/X86-psABI has links to PDFs built from git HEAD revision of the LaTeX source. x86-64 page tables can use 2MB hugepages (and even 1GiB), and Linux does so transparently / opportunistically for anonymous memory, but not file-backed mapping. I'd guess that 4MB is far enough away from 0 that an indexed NULL-pointer deref typically won't index into valid pages. Heh, I see you said the same in your answer. – Peter Cordes Sep 25 '16 at 21:50
  • Why not? Why should another address be chosen? – wildplasser Sep 25 '16 at 22:17

2 Answers2

36

Bottom line: some technical limitations that amd64 has in using large addresses suggest dedicating the lower 2GiB of address space to code and data for efficiency. Thus the stack has been relocated out of this range.

In i386 ABI1

  • stack is located before the code, growing from just under 0x8048000 downwards. Which provides "a little over 128 MB for the stack and about 2 GB for text and data" (p. 3-22).
  • Dynamic segments start at 0x80000000 (2GiB),
  • and the kernel occupies the "reserved area" at the top which the spec allows to be up to 1GiB, starting at at least 0xC0000000 (p. 3-21) (which is what it typically does).
  • The main program is not required to be position-independent.
  • An implementation is not required to catch null pointer access (p. 3-21) but it's reasonable to expect that some of the stack space above 128MiB (which is 288KiB) will be reserved for that purpose.

amd64 (whose ABI is formulated as an amendment to the i386 one (p. 9)) has a vastly bigger (48-bit) address space but most instructions only accept 32-bit immediate operands (which include direct addresses and offsets in jump instructions), requiring more work and less efficient code (especially when taking instruction interdependency into consideration) to handle larger values. Measures to work around these limitations are summarized by the authors by introducing a few "code models" they recommend to use to "allow the compiler to generate better code". (p. 33)

  • Specifically, the first of them, "Small code model", suggests using addresses "in the range from 0 to 231-224-1 or from 0x00000000 to 0x7effffff" which allows some very efficient relative references and array iteration. This is 1.98GiB which is more than enough for many programs.
  • "Medium code model" is based on the previous one, splitting the data into a "fast" part under the above boundary and the "slower" remaining part which requires a special instruction to access. While code remains under the boundary.
  • And only the "large" model makes no assumptions about sizes, requiring the compiler "to use the movabs instruction, as in the medium code model, even for dealing with addresses inside the text section. Additionally, indirect branches are needed when branching to addresses whose offset from the current instruction pointer is unknown." They go on to suggest splitting the code base into multiple shared libraries since these measures do not apply for relative references with offsets that are known to be within bounds (as outlined in "Small position independent code model").

Thus the stack was moved to under the shared library space (0x80000000000, 128GiB) because its addresses are never immediate operands, always referenced either indirectly or with lea/mov from another reference, thus only relative offset limitations apply.

The above explains why the loading address was moved to a lower address. Now, why was it moved to exactly 0x400000 (4MiB)? Here, I came empty so, summarizing what I've read in the ABI specs, I can only guess that it felt "just right":

  • It's large enough to catch any likely incorrect structure offset, allowing for larger data units that amd64 operates on, yet small enough to not waste much of the valuable starting 2GiB of address space.
  • It's equal to the largest practical page size to date and is a multiple of all other virtual memory unit sizes one can think of.

1Note that actual x32 Linuxes have been deviating from this layout more and more as time goes. But we're talking about the ABI spec here since the amd64 one is formally based on it rather than any derived layout (see its paragraph for citation).

Community
  • 1
  • 1
ivan_pozdeev
  • 33,874
  • 19
  • 107
  • 152
0

Static code/data at low addresses, stack at high addresses, is the traditional model. x86-64 follows that; i386 was the unusual one. (With "the heap" in the middle, even though that's not a real thing in asm; there's .data/.bss above .text, brk adding more space just past .bss, and mmap picking random addresses in between.)

The i386 layout left room to put the stack below code, but modern Linux didn't do that anyway. You still get stack addresses like 0xffffe000 in 32-bit code (e.g. under a 64-bit kernel). I'm not sure where a modern build of a 32-bit kernel would put user-space stacks. Of course that's just for the main thread's stack; stacks for new threads have to be allocated manually, usually with mmap.

Why 0x400000 (4 MiB) specifically for the ld default base address?

High enough to avoid mmap_min_addr (default 64k) and leave a gap so NULL deref is still likely to fault noisily, instead of silently read code. Even if it's like ptr[i] with some large i. But otherwise near the bottom of virtual address space is a good place,

Also to optimize the page tables: they're a sparse radix tree (diagram in this answer). Ideally the pages in use share as many higher levels of the tree as possible, so higher levels of the tree have mostly "not present" entries. Less for the kernel to allocate & manage, and the HW page-table walker can internally cache higher level entries (PDE cache) to speed up TLB misses in 4k pages when they're in the same 2M, 1G, or 512G region. And the page-walker(s) accesses memory through cache, so smaller page tables also mean less cache footprint from those accesses.

0x400000 = 4MiB. It's the start of a 2MiB group of pages near the start of the low 1GiB of virtual address space. So an executable with larger code and/or static data that needs multiple pages will have them all in the same subtree of the page tables, touching as few as possible different 1G and 2M regions.

Well, almost as few 1G regions as possible: starting at 0x40000000 (1 GiB) would have put it at the very start of a 1GiB region, not skipping the first two 2MiB largepages of it. But that only matters if your static data size was just below 1GiB, otherwise you still fit in the first 1GiB hugepage region, or extended into the 2nd one anyway.

Basically a duplicate of Why Linux/gnu linker chose address 0x400000? - when I answered that, I forgot I'd already answered this.

Peter Cordes
  • 328,167
  • 45
  • 605
  • 847