Truncate memory mapped file

Question

I am using memory mapped IO for an index file, but the problem is that I'm not able to resize the file if it is mostly empty.

Somewhere before:

MappedByteBuffer map = raf.getChannel().map(MapMode.READ_WRITE, 0, 1 << 30);
raf.close();
// use map
map.force();
map = null;

Resize:

for (int c = 0; c < 100; c++) {
    RandomAccessFile raf = new RandomAccessFile(indexFile, "rw");
    try {
        raf.setLength(newLen);
        if (c > 0) LOG.warn("used " + c + " iterations to close mapped byte buffer");
        return;
    } catch (Exception e) {
        System.gc();
        Thread.sleep(10);
        System.runFinalization();
        Thread.sleep(10);
    } finally {
        raf.close();
    }
}

When using Windows or Linux 32-bit I often have the unmapping problem, but in the 64 bit Linux production environment everything seems to work without warnings, but the file keeps the original size.

Can anyone explain why this happens and/or how to solve the problem?

I fear the problem somehow depends on NFS, caching or timing as it seems to have solved without real intervention (just added logging and waiting and now it works). Even the files which appeased huge after truncationg and which have not been touched since then now have the correct size. Maybe the logging of the new file size after truncate updates some nfs cache. — rurouni, May 13 '11 at 11:09
Problem being discussed is similar to [How to unmap a file](http://stackoverflow.com/questions/2972986), in particular, see [bug#4724038](http://bugs.sun.com/view_bug.do?bug_id=4724038). — dma_k, May 21 '12 at 12:50

score 8 · Accepted Answer · answered Jul 06 '11 at 18:51

Your issue is that you are using unreliable method to close mapped byte buffer (one hundred calls to System.gc() and System.runFinalization() don't guarantee you anything). Unfortunately there is no reliable method in Java API to do that, but on Sun JVM (and perhaps on some others too) you can use the following code:

public void unmapMmaped(ByteBuffer buffer) {
  if (buffer instanceof sun.nio.ch.DirectBuffer) {
    sun.misc.Cleaner cleaner = ((sun.nio.ch.DirectBuffer) buffer).cleaner();
    cleaner.clean();
  }
}

Of course it is JVM-dependent and you should be ready to fix your code if Sun ever decides to change sun.nio.ch.DirectBuffer or sun.misc.Cleaner in an incompatible manner (but actually I don't believe this will ever happen).

score 3 · Answer 2 · answered May 29 '12 at 15:24

This is just a supplement to the previous answer, which is completely correct.

JDK 1.7 complains about the use of sun.misc.Cleaner, saying that classes in this namespace are not a formal part of the JDK, and may disappear in the future. However, as of 1.7 they are still present.

If the .clean() method is unavailable, then using System.gc() can be used as a fallback method, however this must be acknowledged to be a "hack" and care must therefore be used.

While System.gc() cannot force an unreferenced mapping to be closed, in practice it will often cause cleanup to happen. Experience on 32-bit Linux (and Solaris) shows buffers being released during every test during either the first or second call to System.gc(). However, the behavior on Windows is different. In most cases, all mappings are released by the end of the second call to System.gc(), but sometimes it requires 3 calls. There are still occasions where more calls are required, with a requirement for a higher number of calls diminishing in frequency. This can be deceptive, in that tests may indicate that 4 calls are all that is required, only to have it fail on you a month later. 5 calls may then seem adequate, only to lead to failure in 6 months.

Testing to see if a map has been released can be done by using a try/catch block around FileChannel.truncate(), with a loop to re-attempt the operation on failure. The loop cannot be infinite, as there are pathological cases where a particular heap configuration will lead the garbage collector to never clean up a mapping. However, a loop of about 10 will cover almost all cases. If the object isn't gone by that point, then it's not going anywhere and the application will have to give up. That may seem inadequate, but in practice, it is extremely unlikely, and will only be an issue on a JVM that doesn't support cleaners.

Truncate memory mapped file

2 Answers2

Related