10

Memory mapping a large file on Android in Java works good. But when mapping more than ~1.5GB in total even with multiple mapping calls it fails with: 

mmap failed: ENOMEM (Out of memory)

See the full discussion here. Note: It does not fail on a server Linux. The android:largeHeap="true" is enabled for the application.

The following Java code is called a few hundred times requesting ~1MB per call:

ByteBuffer buf = raFile.getChannel().map(allowWrites ? FileChannel.MapMode.READ_WRITE : FileChannel.MapMode.READ_ONLY, offset, byteCount);

to avoid requesting one large contiguous memory chunk which is often harder to be found. See the full code here. Keep in mind that doubling the 'segment size' (i.e. the size of a single map call) has no effect, which means it stops at the similar memory position. Also it is important to note that 2 apps with both slightly under the limit are executing fine (hinting to a per process limit).

Related questions are here, here, here, here, here, here, here and here.

Will using multiple files instead of one file with multiple mappings help?

I've read that this could be a per process limit for the virtual address space. Where can I find more about this? Can I change this setting with NDK e.g. how to call ulimit? Could madvise help me a bit here?

Update

See my answer here for a mmap tool usable in Java

Community
  • 1
  • 1
Karussell
  • 17,085
  • 16
  • 97
  • 197
  • The only solution is probably to implement an upstream cache and call `mapIt` on demand. But the problem is how to unmap those parts as it is hackish with desktop java ('mmap cleaner') and not possible with Android? – Karussell Jul 11 '16 at 19:39
  • Asked this here: http://stackoverflow.com/questions/38315292/unmapping-or-release-a-mappedbytebuffer-under-android – Karussell Jul 12 '16 at 10:01

3 Answers3

8

Your problem is surely caused by virtual address space exhausting. Probably your problem reproduces on 32-bit Android devices, where available to user address space is physically limited to 2GB and cannot be bumped. (Although it may be 3GB (unlikely so) and it is configured during OS build process). Probably ~500 MB is used for system libraries, JVM and its heap. And ~1.5 GB is available for you.

The only way in this situation IMO - keep being mapped only file sections that are really used now and unmap unused ones as soon as possible. You can utilize some kind of sliding window where only small part of file will be mapped to memory, and when you finish - unmap that part, advance your window position and map that updated window, so on.

Also when you map whole large file - your process becomes an attractive victim for system's Out-Of-Memory killer. Because when you read such mapped file - consumption of physical memory raises and at some moment process will be killed.

Sergio
  • 8,099
  • 2
  • 26
  • 52
  • From user reports it shows up on 64bit systems as well, but maybe there is some static setting in Android. Your suggested solution is not trivial under Android as explicit unmapping is not supported, or do you know a tool for that? See my other question http://stackoverflow.com/questions/38315292/unmapping-or-release-a-mappedbytebuffer-under-android Furthermore why do you think it will raise physical memory usage? Memory usage should be negligible small IMO especially if you map the whole file – Karussell Jul 22 '16 at 08:35
  • 1
    You can check address space size limit with `getrlimit(RLIMIT_AS)` (only for native code, seems like java has no ways to accomplish such task). Next, about physical memory usage. When you map file into memory in general it will take as much physical memory as you have mapped. But you may not see it immediately because mapping may be created in non-blocking way, without read-ahead and complete memory allocation. Data will be mapped to new pages on demand. It is done by kernel and is transparent for user. Try reading sequentially from mapping byte-by-byte - you'll see memory usage increasing. – Sergio Jul 22 '16 at 09:11
  • Yes, seems like implicit unmapping via Java API is impossible. Then it may be reasonable to implement your own mapping/unmapping native module that will use `mmap()` / `munmap()` and deliver mapped data via [direct byte buffers](https://docs.oracle.com/javase/8/docs/technotes/guides/jni/jni-14.html#NewDirectByteBuffer). It should be effective approach. And gives more flexibility to you in what you want to map and when. – Sergio Jul 22 '16 at 09:20
  • I understand now better what you mean regarding the physical memory. But isn't the OS allowed to free memory if it would be too much? I thought you can map regions a lot larger than the physical memory. Such a native module would be wonderful to have ... do you know some open source tools/helpers in this direction :) ? – Karussell Jul 22 '16 at 13:48
  • Mapping size can not be greater that physical RAM + swap (if any). Also it can not be greater than process address space limit (forced via `setrlimit(RLIMIT_AS)`). OS won't free anything without mapping as far as I know. But I guess such approach is technically possible in order to overcome physical RAM size limit but not virtual address space restriction. – Sergio Jul 22 '16 at 14:28
  • About module: I haven't heard about such things, but anyway it shouldn't be too hard to implement it by yourself. But you should use it carefully. E.g. you decided to use direct byte buffers in java that wrap mapping pointer that was created at native. Then after unmapping you can not use that `ByteBuffer` anymore since it refers to invalid address. You need some sort of control at java level. Probably this problem is one of the reasons why Java API doesn't expose such facility as explicit unmapping. – Sergio Jul 22 '16 at 14:29
  • See this article where it is described that it is possible to simulate very large RAM: https://dzone.com/articles/using-memory-mapped-file-huge Also I know that I've used my software on a <1gb RAM phone with a mapping of >1gb but anyway sad state of mmap on Android via Java ... – Karussell Jul 22 '16 at 15:00
1

As we cannot increase the virtual address limit on Android via an API (that does not need root access), also I've not yet seen this in the Android source code. The only possible solution I see is to implement kind of a cache, which mmaps segments on access and releases older segments if a certain number of segments is already mmapped. This means we are doing the work the OS normally does for us automatically, which is a bit ugly.

To make it working under Android one can use this answer / util-mmap. Hopefully someone can implement such a mmap cache for Android at some point, maybe even us :)

Community
  • 1
  • 1
Karussell
  • 17,085
  • 16
  • 97
  • 197
0

try to add largeHeap in your manifest. May be it works

parik dhakan
  • 787
  • 4
  • 19