Prevent OutOfMemory when using java.nio.MappedByteBuffer

Question

Consider application, which create 5-6 threads, each thread in cycle allocate MappedByteBuffer for 5mb page size.

MappedByteBuffer b = ch.map(FileChannel.MapMode.READ_ONLY, r, 1024*1024*5);

Sooner or later, when application works with big files, oom is thrown

java.io.IOException: Map failed  at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:758)
Caused by: java.lang.OutOfMemoryError: Map failed
        at sun.nio.ch.FileChannelImpl.map0(Native Method)
        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:755)

According to specification, MappedBuffer should dispose direct memory as soon as it is GC itself. Looks like the problem is, that MappedBuffer-s are GC-ed too late, later then direct memory finished.

How to avoid this situation ? Probably say MappedBuffer to dispose implicitly or use some kind of pool of MappedBuffer

The exception is **NOT OOM** but IOException. You are running out of virtual address space. Show quite a bit more code. Mapped buffers and reclamation in java is a long time issues (still unresolved elegantly) — bestsss, Dec 19 '11 at 06:22

score 6 · Answer 1 · answered Dec 18 '11 at 22:10

You can avoid having to trigger a GC by cleaning up the mapped byte buffers directly.

public static void clean(ByteBuffer bb) {
    if(bb == null) return;
    Cleaner cleaner = ((DirectBuffer) bb).cleaner();
    if(cleaner != null) cleaner.clean();
}

Provided you call this before discarding, you won't run out of virtual memory.

Perhaps you can look at creating larger ByteBuffers less often (unless you have a large number of files) Creating a MappedByteBuffer is not free (takes about 50 micro-seconds on some machines)

Unfortunately, these methods appear to do something but they don't actually unmap the ByteBuffer as such. This method only works for direct memory. — Peter Lawrey, Jan 24 '13 at 13:54

score 3 · Answer 2 · edited May 23 '17 at 12:24

3

The error message says "map failed", not "heap space" or "permgen space". This means the JVM doesn't have enough address space available.

See this bug in Sun's database, and also this question.

The first link provides a workaround (ewww) which is close the what the second link says:

    try {
        buffer = channel.map(READ_ONLY, ofs, n);
    } catch (java.io.IOException e) {
        System.gc();
        System.runFinalization();
        buffer = channel.map(READ_ONLY, ofs, n);
    }

edited May 23 '17 at 12:24

Community

1
1

answered Dec 18 '11 at 17:03

fge

119,121
33
254
329

Depending on the GC algorithms and its parameters the `System.gc()` call might be a NOP. Using `-XX:+DisableExplicitGC` effectively removes the ugly patch of catch the OOM and retrying after the GC request – bestsss Dec 19 '11 at 06:42

duffymo · Answer 3 · 2011-12-18T17:59:05.970

2

Maybe a WeakHashMap to pool those MappedBuffers would work.

But before you guess about the root cause, I'd recommend hooking your app up to Visual VM 1.3.3, with all the plugins installed, so you can see exactly what's causing the OOM error. You're presuming that these MappedBuffers are doing it, but they're only 5MB each for 5-6 threads - 25-30MB total.

Better to have data than guess. Visual VM will get it for you.

edited Dec 18 '11 at 17:59

answered Dec 18 '11 at 16:47

duffymo

305,152
44
369
561

Well actually, each thread can create a lot of buffer in cycle, depends on file size. This mean, that the total amount of buffers are not limited. Probably i should use some kind of semaphore to prevent allocation of huge amount of memory. Anyway i will try profiler, you suggested – user12384512 Dec 18 '11 at 16:53
Mapped buffers cannot be pooled! They are created to cover an address returned by the mapping virtual address to a location in file and loading data by page faults + writing back on file handle close or request. – bestsss Dec 19 '11 at 06:20
1

...and `WeakHashMap` is totally useless/harmful since the key won't be the Buffer itself, hence it won't be even a GC-available before the `WeakHashMap` is expunged (internally). Using a GC dependent structure for the problem is definitely wrong. – bestsss Dec 19 '11 at 06:27
Funny how bestsss has commented on every answer but offered nothing himself. Let's see your answer to the question. – duffymo Dec 19 '11 at 10:13
I'd answer, I asked for more code. With the one currently provided there is no solution. It runs out of virtual memory and map may not succeed. It's a hard one and usually requires design changes and thinking ahead. You can read my comment below the question. There is no offense towards your answer, yet any GC related solution is bound to be worse than the bug, itself. GC just fails to reclaim the virtual memory (by garbaging all views of the address quickly enough) and if the explicit GC is disabled... – bestsss Dec 19 '11 at 12:41
No offense taken; I think your comments are spot on and most insightful. Looking at your profile I don't see as many answers as I'd expect from someone with the knowledge you seem to have. – duffymo Dec 19 '11 at 13:37
but... i write comments most of the time (well obviously). Or when I answer I put quite a bit of code (sometimes over 200 lines) – bestsss Dec 21 '11 at 01:33
I found your comments here to be very good, indeed. Thank you for posting them - very educational. – duffymo Dec 21 '11 at 01:40

score 0 · Answer 4 · answered Dec 19 '11 at 05:51

0

MappedBuffer should dispose direct memory as soon as it is GC itself

It doesn't actually say that anywhere that I can see. There is a long-standing Bug Parade item that says it is never released.

It does say this:

It is therefore recommended that direct buffers be allocated primarily for large, long-lived buffers

answered Dec 19 '11 at 05:51

user207421

305,947
44
307
483

1

They are disposed for sure, the main issue is that a tiny java object holds reference to megabytes and the GC doesn't deem it anywhere urgent to releases and finalize/unmap the virtual memory. – bestsss Dec 19 '11 at 06:24
@bestsss Either they are 'disposed' or they aren't. Make up your mind. You need to look up Bug Parade item #4724038. It doesn't agree with you. – user207421 Dec 19 '11 at 09:09
2

read the comment again. they are disposed but too often much delayed. I know the bug well and the native impl on both Windows and Linux/Solaris. The bug doesn't tell the mapped buffers are not unmapped, they are AFTER the ByteBuffer (and its views) are garbage collected. They just cannot be forcibly closed/unmapped via normal means. It has been always possible to hack around and unmap the Buffer risking SIGSEV or using another file, or any undefined result. – bestsss Dec 19 '11 at 09:32
1

the bug has no simple solution b/c an address cannot be atomically unmapped (+flush) and then remapped to NUL device. OS providing such functionality would do the trick. Adding synchronizations would kill performance. If an OS provide m_null_unmap or something alike it'd do the trick. That's it: marks the pages not owned by the process, flushes them and maps it back to null device. Any concurrent access will lead to page fault, so the OS can handle it. – bestsss Dec 19 '11 at 09:39

Prevent OutOfMemory when using java.nio.MappedByteBuffer

4 Answers4

Linked