0

The function call to mmap64() is as follows:

addr = (unsigned char*) mmap64(NULL, regionSize, PROT_READ|PROT_WRITE, MAP_SHARED, FileDesc, (unsigned long long)regionAddr);

The arguments typically have values as follows:

regionSize = 0x20000;
FileDesc   = 27;
regionAddr = 0x332C0000;

Obviously in the code these values are not hard-coded like that, but I just want to show you what the typical values for them will be.

The problem:

mmap64() call works perfectly in Red Hat Linux 6.6, kernel version: 2.6.32-504.16.2.el6.x86_64. It fails in Red Hat Linux 7.2, kernel version: 3.10.0-327.13.1.el7.x86_64.

No difference in code as far as I'm aware of.

The errno returned is "Invalid argument" or errno #22 (EINVAL). Looking at this reference http://linux.die.net/man/3/mmap64, I see 3 possiblilities for the EINVAL error:

  1. We don't like addr, length, or offset (e.g., they are too large, or not aligned on a page boundary). -> most likely culprit in my situation.
  2. (since Linux 2.6.12) length was 0. -> Impossible, I checked length (regionSize) value in debug print, it was 0x20000.
  3. flags contained neither MAP_PRIVATE or MAP_SHARED, or contained both of these values. -> Can't be this as you can see from my function call, only MAP_SHARED flag was given as argument.

So I'm stuck at the moment. Not sure how to debug this. This problem is 100% reproducible. Anyone have tips on what could have changed between the two OS versions to cause this?

Splaty
  • 604
  • 1
  • 8
  • 21
  • 1
    Why not simply use `mmap()` without the suffix if you're building 64-bit executables? Does that make any difference to your problem? Are you sure there's nothing already allocated to the region you specify? What would happen if you let `mmap64()` choose where to place the memory? – Jonathan Leffler Jul 19 '16 at 21:31
  • 3
    The last argument is called `offset` in the synopsis ([`mmap64()`](http://linux.die.net/man/3/mmap64)), and: _`offset` must be a multiple of the page size as returned by `sysconf(_SC_PAGE_SIZE)`._ Is your value a multiple of the page size? It looks to me like it has too few trailing zeros in the hex (a multiple of 512, not of 4K or larger). – Jonathan Leffler Jul 19 '16 at 21:40

1 Answers1

3

Transferring comments into an answer (with some comments trimmed as not relevant).

Why not simply use mmap() without the suffix if you're building 64-bit executables? Does that make any difference to your problem?

However, I think your problem is what you call regionAddr. The last argument to mmap64() is called offset in the synopsis, and:

offset must be a multiple of the page size as returned by sysconf(_SC_PAGE_SIZE).

Is your value of regionAddr a multiple of the page size? It looks to me like it has too few trailing zeros in the hex (it is a multiple of 512, but not a multiple of 4K or larger).

Note that the question originally had a different value on display for regionAddr — see also the comments below.

regionAddr = 0x858521600;

and

addr = (unsigned char*) mmap64(NULL, regionSize, PROT_READ|PROT_WRITE, MAP_SHARED, FileDesc, (unsigned long long)regionAddr);

With the revised information (that the value in regionAddr is 0x332C0000 or decimal 828521600), it is less than obvious what is going wrong.

Jonathan Leffler
  • 730,956
  • 141
  • 904
  • 1,278
  • Thanks for the tip. sysconf(_SC_PAGE_SIZE) = 4096. The offset is 789315584 (0x2F0C0000) on first try, 853278720 (0x32DC0000) on second try. It is a multiple of the page size (unless my math is wrong). I'm digging deeper into the offset because it looks like the only thing that could go wrong here. – Splaty Jul 20 '16 at 14:38
  • 1
    That's not what the question says: `regionAddr = 0x858521600;` and `addr = (unsigned char*) mmap64(NULL, regionSize, PROT_READ|PROT_WRITE, MAP_SHARED, FileDesc, (unsigned long long)regionAddr);`. You may be correct that your real code should be OK (both `0x2F0C0000` and `0x32DC0000` are aligned on 256 KiB boundaries), but we can't help you if you don't show us the real code. – Jonathan Leffler Jul 20 '16 at 14:41
  • Yeah, I made a mistake in my original question. regionAddr should be decimal 858521600 (0x332C0000). About the real code, it's complicated and proprietary. I don't have a good idea of how to organize it into a good question. I'm going to dig deeper at the offset argument, as I believe that is the only argument that could be going wrong here. Is there a way for mmap() to give a more helpful error for debugging other than "Invalid argument"? – Splaty Jul 20 '16 at 18:53
  • 1
    OK; mistakes happen, but it is hard for people to help you reliably when the information in the question is (accidentally) unreliable. The type of the last argument is, formally, `off_t` — are you sure that's equivalent to `unsigned long long`? If not, it might account for the trouble. I also note that the man page has `mmap64()` in the title but plain `mmap()` in the synopsis. Have you tried replacing `mmap64()` with `mmap()` — and did it make any difference. I think I'd be surprised if it made a difference, but we need to rule out the improbable too. Consider an MCVE ([MCVE]). – Jonathan Leffler Jul 20 '16 at 19:39
  • Thanks Jonathan for the help. I think I've found the problem. It's due to the file descriptor argument. We are trying to memory map to /dev/mem which has been protected by the kernel. The exact same problem occured for this poster here: http://stackoverflow.com/questions/11891979/accessing-mmaped-dev-mem. I guess the difference in kernel versions had the different kernel mapping protection implementation. Thanks for all the help. – Splaty Jul 21 '16 at 15:53
  • 1
    Interesting. You weren't getting EBADF but the problem was related to the descriptor and what it referenced. Evidently, I should have asked about which file. If you'd mentioned the file, someone else might have spotted the problem sooner. Glad it is solved what's going wrong. Do you have a workaround yet? – Jonathan Leffler Jul 21 '16 at 15:57
  • Yup, memory mapping to a location other than /dev/mem solved the issue. Might cause other problems with diagnostic tools still looking for the mapping in /dev/mem but glad we found this issue because it is a bug that would have occurred sooner or later with the (not so) new mapping protection. – Splaty Jul 22 '16 at 12:52