2

I tried to do an mmap syscall after switching to 64 bit mode on x86 Linux (Linux version 5.15.6), but the syscall returns ENOMEM. A minimal reproducible example is given by the following assembly code:

.text
.global _start

_start:
.code32

jmp $0x33, $start64 // jmp to start64 and change mode to 64 bit

.code64
start64:

mov $9, %rax // mmap
mov $0, %rdi // NULL
mov $0x1000, %rsi // size
mov $1, %rdx // PROT_READ
mov $0x22, %r10
mov $0, %r8
mov $0, %r9
syscall

mov $60, %rax // 64 bit style exit
mov $0, %rdi
syscall

The corresponding output of strace is:

execve("./mmap_test32", ["./mmap_test32"], 0x7ffe0ccd7920 /* 60 vars */) = 0
mmap(NULL, 4096, PROT_READ, MAP_PRIVATE|MAP_ANONYMOUS, 0, 0) = -1 ENOMEM (Cannot allocate memory)
exit(0)                                 = ?
+++ exited with 0 +++

Does anyone have an idea why this is not working?

Peter Cordes
  • 328,167
  • 45
  • 605
  • 847
rgba
  • 51
  • 2
  • 1
    Related (but not duplicate): https://stackoverflow.com/questions/24113729/switch-from-32bit-mode-to-64-bit-long-mode-on-64bit-linux ... this is very poorly supported, and I can't be sure that either the kernel or strace is actually doing anything sensible. – o11c Dec 08 '21 at 21:30
  • What we know for sure is that the syscall returns `-1`, so it is pretty certain that it fails. Other syscalls (like `open`) do work properly. I am aware that this mode is poorly supported, but it would be interesting to know what the reasons are and if there is any possible fix to it. – rgba Dec 08 '21 at 21:41
  • 2
    wild speculation: does `MAP_32BIT` help? I'm imagining what might happen if it generates a random unused 64-bit address then tries to add it to a 32-bit VM ... – o11c Dec 08 '21 at 23:02
  • 1
    Indeed, when adding the `MAP_32BIT` flag the `mmap` does work correctly. Thanks! However, it would be nice to be able to either increase the size of the virtual memory or prevent `mmap` from generating too big addresses. The reason is that we would like to execute unaltered, foreign code afterwards. – rgba Dec 09 '21 at 00:29
  • 2
    IDK if it's possible to reset what kind of process Linux things this is. Possibly with a `personality(2)` system call? If that can unset `ADDR_LIMIT_32BIT`? Maybe with `personality(PER_LINUX)` instead of `PER_LINUX_32BIT` which is probably what execve on a 32-bit ELF implies. Obviously it would be easier if you could just make this a 64-bit process in the first place, or execve a 64-bit ELF instead of doing major hacks like hard-coding the kernel's 64-bit user-CS segment selector constant and changing modes in user-space. – Peter Cordes Dec 09 '21 at 01:08
  • The personality of the process is `PER_LINUX|READ_IMPLIES_EXEC`, setting this to `PER_LINUX` doesn't change anything. Generally, it doesn't seem like setting the personality has an effect on the running process, for example after setting `ADDR_NO_RANDOMIZE` in a normal 64 process doesn't prevent the resulting address of an mmap to be random, but using `setarch -R ./test` (which does a personality syscall and then execve) does. – rgba Dec 12 '21 at 00:38
  • I also didn't see `ADDR_LIMIT_32BIT` (set with `setarch -B ./test`) having any effect, neither for our 32 bit to 64 bit switch process nor for a regular 64 bit process, which is a bit odd. I also had a look at at the modify_ldt syscall, but this also doesn't seem to help, so i will treat this experiment as failed. – rgba Dec 12 '21 at 00:38

0 Answers0