6

In answer to a question about mapping non-contiguous blocks of files into contiguous memory, here, it was suggested by one respondent that I should use VirtualAllocEx() with MEM_RESERVE in order to establish a 'safe' value for the final (lpBaseAddress) parameter for MapViewOfFileEx().

Further investigation revealed that this approach causes MapViewofFileEx() to fail with error 487: "Attempt to access invalid address." The MSDN page says:

"No other memory allocation can take place in the region that is used for mapping, including the use of the VirtualAlloc or VirtualAllocEx function to reserve memory."

While the documentation might be considered ambiguous with respect to valid sequences of calls, experimentation suggests that it is not valid to reserve memory for MapViewOfFileEx() using VirtualAllocEx().

On the web, I've found examples with hard-coded values - example:

#define BASE_MEM     (VOID*)0x01000000

...

hMap = MapViewOfFileEx( hFile, FILE_MAP_WRITE, 0, 0, 0, BASE_MEM );

To me, this seems inadequate and unreliable... It is far from clear to me why this address is safe, or how many blocks can be safely be mapped there. It seems even more shaky given that I need my solution to work in the context of other allocations... and that I need my source to compile and work in both 32 and 64 bit contexts.

What I'd like to know is if there is any way to reliably reserve a pool of address space in order that - subsequently - it can be reliably used by MapViewOfFileEx to map blocks to explicit memory addresses.

Community
  • 1
  • 1
aSteve
  • 1,956
  • 1
  • 20
  • 35
  • There is not. Only thing you can do is do it early. – Hans Passant Aug 25 '12 at 12:26
  • @Hans, while I've no reason to disbelieve you, I would welcome a definitive reference stating this. It seems silly to have VirtualAllocEx support for explicit addresses, but no way to reliably reserve address space. Mapping early might work in some contexts, but not for multi-threaded programs where different views need to be mapped during the lifetime of the process. – aSteve Aug 25 '12 at 12:48
  • The definite reference is the MSDN article, understood that you don't like what it says. Clearly you have no hope of mapping a view to a specific address reliably when you cannot control other code that allocates virtual memory. The logical conclusion is thus that you should not try to make it work and leave it up to the OS to pick an address. – Hans Passant Aug 25 '12 at 13:20
  • The MSDN doesn't say there's no way to do this reliably - it just says that I shouldn't use VirtualAlloc(). You're right, I don't like what it says... interpreted as you have, it makes memory mapping significantly less flexible under Windows than under Linux/Unix. If that's an unavoidable inherent problem with Windows, I can accept that - but I'd strongly prefer otherwise. – aSteve Aug 25 '12 at 13:37

3 Answers3

3

You almost got to the solution by yourself but fell short of the last small step. As you figured, use VirtualAlloc (with MEM_RESERVE) to find room in your address space, but after that (and before MapViewOfFileEx) use VirtualFree (with MEM_RELEASE). Now the address range will be free again. Then use the same memory address (returned by VirtualAlloc) with MapViewOfFileEx.

MichaelS
  • 5,941
  • 6
  • 31
  • 46
l_belev
  • 31
  • 2
  • 1
    There is an obvious race condition if you have several threads (you may have to retry a few times), but yeah... this is the only way to _safely_ provide a valid address (and range). – Damon Nov 26 '14 at 12:32
  • Oh, there are other possible solutions. For instance you can manually scan your process' address space with VirtualQuery and look for big enough hole. The one with VirtualAlloc is just the simplest. – l_belev Nov 26 '14 at 16:13
  • As for the issue with the multi-threading, yes, you can retry until you get it. But this can get ugly if you do the MapViewOfFileEx at multiple steps (thats what the OP had in mind). Another possible way for this that comes to mind is to have your thread first suspend all other threads in the process, do it's memory mapping and then resume them again. Depending on your particular case, this may not be an overkill if the mapping stuff is done rarely (say only once for the process life time). – l_belev Nov 26 '14 at 16:19
0

If you provide a base address, the function will try to map your file at that address. If it cannot use that base address (because something is already using all or part of the requested memory region), then the call will fail.

For most applications, there's no real point trying to fix the address yourself. If you're a sophisticated database process and you're trying to carefully manage your own memory layout on a machine with a known configuration for efficiency reasons, then it might be reasonable. But you'd have to be prepared for failure.

In 64-bit processes, the virtual address space is pretty wide open, so it might be possible to select a base address with some certainty, but I don't think I'd bother.

From MSDN:

While it is possible to specify an address that is safe now (not used by the operating system), there is no guarantee that the address will remain safe over time. Therefore, it is better to let the operating system choose the address.

I believe "over time" refers to future versions of the OS and whatever run-time libraries you're using (e.g., for memory allocation), which might take a different approach to memory layout.

Also:

If the lpBaseAddress parameter specifies a base offset, the function succeeds if the specified memory region is not already in use by the calling process. The system does not ensure that the same memory region is available for the memory mapped file in other 32-bit processes.

So basically, your instinct is right: specifying a base address is not reliable. You can try, but you must be prepared for failure.

So to directly answer your question:

What I'd like to know is if there is any way to reliably reserve a pool of address space in order that - subsequently - it can be reliably used by MapViewOfFileEx to map blocks to explicit memory addresses.

No, there isn't. Not without applying many constraints on the runtime environment (e.g., limiting to a specific version of the OS, setting base addresses for all of your DLLs, disallowing DLL injection, etc.).

Adrian McCarthy
  • 45,555
  • 16
  • 123
  • 175
  • Some docs assume mapping at an explicit address is to replicate objects at the same locations in two processes sharing memory. I understand this, but I am memory mapping files in single processes - not implementing shared memory. I also understand that it is 'recommended' to allow the OS to allocate the address whenever possible - but, I note, this is not possible if you need to map non-contiguous blocks (from a device) into contiguous memory addresses. I want to know how the "sophisticated" process should manage its address space correctly... 'fingers crossed' seems just a hack. – aSteve Aug 25 '12 at 16:22
0

What you are trying to do is impossible.

From the MapViewOfFileEx docs, the pointer you supply is "A pointer to the memory address in the calling process address space where mapping begins. This must be a multiple of the system's memory allocation granularity, or the function fails."

The memory allocation granularity is 64K, so you cannot map disparate 4K pages from the file into adjacent 4K pages in virtual memory.

arx
  • 16,686
  • 2
  • 44
  • 61
  • I'm aware that different systems have different granularities... I can easily establish this platform-specific detail using GetSystemInfo(). I'd be entirely happy if I were to map non-adjacent 64K pages into adjacent 64K pages in memory. – aSteve Aug 25 '12 at 19:44
  • If you use a pretend page size of 64K to match the allocation granularity then you'll do far more I/O than required. For example, updating a 2K structure straddling a 64K boundary will require you to write 128K rather than 8K. On the other hand, if your structures are very large this is much less of a problem. But this is moot. You've still got the problem that you can't stop the OS stealing your address space (unless you have a process-wide lock so you can "atomically" unreserve or unmap one "page" and map another). – arx Aug 25 '12 at 20:23
  • 1
    I have no problems with granularity-oriented efficiency - with an application page size of 1024K, for example, any plausible system granularity is fine. The strategy works great on Linux - essentially because I can reserve memory by mapping /dev/zero, then map explicitly into the space that initial mapping reserves. Managing the address space is the real problem under Windows. It seems bizarre that there should be no way to reliably reserve address space for explicit-address mapping. – aSteve Aug 25 '12 at 23:33