29

What limits the size of a memory-mapped file? I know it can't be bigger than the largest continuous chunk of unallocated address space, and that there should be enough free disk space. But are there other limits?

MarkusQ
  • 21,814
  • 3
  • 56
  • 68
user88185
  • 293
  • 1
  • 3
  • 4

6 Answers6

28

You're being too conservative: A memory-mapped file can be larger than the address space. The view of the memory-mapped file is limited by OS memory constraints, but that's only the part of the file you're looking at at one time. (And I guess technically you could map multiple views of discontinuous parts of the file at once, so aside from overhead and page length constraints, it's only the total # of bytes you're looking at that poses a limit. You could look at bytes [0 to 1024] and bytes [240 to 240 + 1024] with two separate views.)

In MS Windows, look at the MapViewOfFile function. It effectively takes a 64-bit file offset and a 32-bit length.

Jason S
  • 184,598
  • 164
  • 608
  • 970
9

This has been my experience when using memory-mapped files under Win32:

If your map the entire file into one segment, it normally taps out at around 750 MB, because it can't find a bigger contiguous block of memory. If you split it up into smaller segments, say 100MB each, you can get around 1500MB-1800MB depending on what else is running.

If you use the /3g switch you can get more than 2GB up to about 2700MB but OS performance is penalized.

I'm not sure about 64-bit, I've never tried it but I presume the max file size is then limited only by the amount of physical memory you have.

Alan Clark
  • 2,017
  • 21
  • 28
  • On 64-bit the limit is the available address space, the same as on 32-bit. You can map a 1 GB view on a 32-bit machine which only has 32MB physical memory. The same way you can map a 10TB view on a 32-bit machine which only has 0.5GB of memory. Only the 4kb pages you really access are mapped into real memory. – springy76 Jul 22 '15 at 10:36
  • 2
    @springy76: you cannot map a 10TB view on a 32-bit machine. Only on 64-bit. – Sergey K. Oct 27 '15 at 17:56
  • 5
    @SergeyK. you are damn right, thats also what I wanted to write -- but my fingers typed something else. It should have read "The same way you can map a 10TB view on a 64-bit machine which only has 0.5GB of memory" – springy76 Oct 28 '15 at 08:14
1

Yes, there are limits to memory-mapped files. Most shockingly is:

Memory-mapped files cannot be larger than 2GB on 32-bit systems.

When a memmap causes a file to be created or extended beyond its current size in the filesystem, the contents of the new part are unspecified. On systems with POSIX filesystem semantics, the extended part will be filled with zero bytes.

Even on my 64-bit, 32GB RAM system, I get the following error if I try to read in one big numpy memory-mapped file instead of taking portions of it using byte-offsets:

Overflow Error: memory mapped size must be positive

Big datasets are really a pain to work with.

Community
  • 1
  • 1
JYun
  • 311
  • 2
  • 12
1

The limit of virtual address space is >16 Terabyte on 64Bit Windows systems. The issue discussed here is most probably related to mixing DWORD with SIZE_T.

StSt
  • 19
  • 2
  • 1
    Hello Mark, the information and discussion here is confusing publishing wrong information. It is missguiding readers to assume there is a limitation to 4 gbyte. The maximum limit on windows system for mmaped files is 16 Terabyte. – StSt Jun 04 '20 at 17:51
  • 2
    Hello Mark, thank you for confirmation. Please let me add, that the given number is a theoretical limit only. In general the really available virtual address space is limited by the disk i/o speed versus the hard page fault frequency. This factor defines the maximum usable pagefile size. – StSt Jun 04 '20 at 18:49
  • Would be useful to add a citation. – vy32 Oct 13 '21 at 17:39
  • 1
    On 32-bit Windows the limit is 16TB, and on 64-bit Windows the limit is 256TB. Virtual memory is also physically limited by the available disc space. This is taken from https://stackoverflow.com/questions/6608820/is-virtual-memory-infinite#6609035 – StSt Oct 15 '21 at 05:53
  • 1
    For allocating and managing huge amounts of memory in Windows Systems see also https://www.codeproject.com/Articles/5267559/A-monitored-memory-mapped-std-allocator-for-mass-d – StSt Oct 15 '21 at 05:57
  • how do you possibly allocate 16TB of memory on a 32-bit system? – vy32 Oct 15 '21 at 09:10
  • 1
    For more information on Windows virtual address space please be advised to https://techcommunity.microsoft.com/t5/windows-blog-archive/pushing-the-limits-of-windows-virtual-memory/ba-p/723750 – StSt Oct 17 '21 at 05:24
1

Under Windows: "The size of a file view is limited to the largest available contiguous block of unreserved virtual memory. This is at most 2 GB minus the virtual memory already reserved by the process. "

From MDSN.

I'm not sure about LINUX/OSX/Whatever Else, but it's probably also related to address space.

Furious Coder
  • 1,160
  • 1
  • 11
  • 17
  • 1
    just to clarify, that's the size of a file view, not the size of the file itself. – Jason S Apr 07 '09 at 16:14
  • 5
    "2 GB minus the virtual memory already reserved by the process" is on 32 bit machine? Is it different on a 64 bit box? – user88185 Apr 07 '09 at 16:35
-3

There should be no other limits. Aren't those enough? ;-)

dwc
  • 24,196
  • 7
  • 44
  • 55