Why does mmap fail on iOS?

Question

I'm trying to use mmap to read and play audio files on iOS. It works fine for files up to about 400MB. But when I try a 500MB file, I get a ENOMEM error.

char *path = [[[NSBundle mainBundle] pathForResource: @"test500MB" ofType: @"wav"] cStringUsingEncoding: [NSString defaultCStringEncoding]];
FILE *f = fopen( path, "rb" );
fseek( f, 0, SEEK_END );
int len = (int)ftell( f );
fseek( f, 0, SEEK_SET );

void *raw = mmap( 0, len, PROT_READ, MAP_SHARED, fileno( f ), 0 );

if ( raw == MAP_FAILED ) {
    printf( "MAP_FAILED. errno=%d", errno ); // Here it says 12, which is ENOMEM.
}

Why?

I'd be happy with an answer like "700MB is the virtual memory limit, but sometimes the address space is fragmented, so you DO get 700MB but in smaller chunks". (This is just speculation, I still need an answer)

The Apple doc page about virtual memory says:

Although OS X supports a backing store, iOS does not. In iPhone applications, read-only data that is already on the disk (such as code pages) is simply removed from memory and reloaded from disk as needed.

which seems to confirm that mmap should work for blocks larger than the physical memory but still doesn't explain why I'm hitting such a low limit.

Update

• This answer is interesting, but 500MB is well below the 700MB limit it mentions.
• This discussion mentions contiguous memory. So memory fragmentation could be a real issue?
• I'm using iPod Touch 4th generation which has 256MB physical memory.
• The point of my research is to see if there's a better way of managing memory when loading read-only data from files than "keep allocating until you get a memory warning". mmap seemed like a nice way to solve this...

Update 2

I expect mmap to work perfectly with the new 64bit version of iOS. Will test once I get my hands on a 64bit device.

Answer 1

After further investigation and reading this excellent blog post by John Carmack, here are my conclusions:

• 700MB is the limit for virtual memory on iOS (as of 2012, 32-bit iOS)
• It may or may not be available in a single block; this depends on device state and app behaviour

Therefore, to reliably mmap 700MB worth of file data it is necessary to break it into smaller chunks.

Answer 2

I don't have an answer for you, but I did test your code on my iPhone 5 running 6.0.1 and mmap succeeded on a 700MB ISO file. So I would start with other factors and assume mmap is working properly. Perhaps the error you're getting back is not really due to memory, or perhaps the memory on the device itself is somehow exhausted to where mmap is failing; try rebooting the device. Depending on your version of iOS, I wonder also if perhaps your seek to the end of the file might be causing mmap to try and map the entire file in; you might try cleaning that up and use stat instead to determine the file size, or close then re-open the file descriptor before mapping. These are all just ideas; if I could reproduce your error, I'd be glad to help fix it.

Answer 3

Use NSData and dont touch mmap directly here.

To get the advantages of faulting reads, use NSDataReadingMapped. NSData ALSO frees bytes when in low-mem situations

Answer 4

Normally the amount of physical memory available has nothing to do with whether or not you are able to mmap a file. This is after all VIRTUAL memory we are talking about. The problem on iOS is that, at least according to the iOS App Programming Guide, the virtual memory manager only swaps out read-only sections... In theory that would mean that you are not only constrained by the amount of available address space, but you are also constrained by the amount of available RAM, if you are mapping with anything other than PROT_READ. See http://developer.apple.com/library/ios/#documentation/iphone/conceptual/iphoneosprogrammingguide/TheiOSEnvironment/TheiOSEnvironment.html

Nevertheless, it may well be that the problem you are having is a lack of contiguous memory large enough for your mapping in the virtual address space. So far as I can find, Apple does not publish the upper memory limit of a user-mode process. Typically the upper region of the address space is reserved for the kernel. You may only have 16-bits of memory to work with in user-mode.

What you can't see without dumping a memory map in the debugger, is that there are many (I counted over 100 in a simple sample application from Apple) shared libraries (dylibs) loaded into the process address space. Each of these are also mmap'd in, and each will fragment the available address space.

In gdb you should be able to dump the memory mappings with 'info proc mappings'. Unfortunately in lldb the only equivalent I've been able to find is 'image list', which only shows shared library mappings, not data mmap mappings.

Using the debugger in this way you should be able to determine if the address space has a contiguous block large enough for the data you are trying to map, though it make take some work to discover the upper limit (Apple should publish this!)