httpasyncclient OutOfMemoryError Direct buffer memory

Question

i use httpasyncclient 4.1, i found that the direct memory continuously growing. and finally throw java.lang.OutOfMemoryError.

Exception in thread "I/O dispatcher 2" java.lang.OutOfMemoryError: Direct buffer memory
    at java.nio.Bits.reserveMemory(Bits.java:658)
    at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
    at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
    at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:174)
    at sun.nio.ch.IOUtil.read(IOUtil.java:195)
    at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
    at org.apache.http.impl.nio.codecs.AbstractContentDecoder.readFromChannel(AbstractContentDecoder.java:129)
    at org.apache.http.impl.nio.codecs.LengthDelimitedDecoder.read(LengthDelimitedDecoder.java:86)
    at org.apache.http.nio.util.SimpleInputBuffer.consumeContent(SimpleInputBuffer.java:68)
    at org.apache.http.nio.protocol.BasicAsyncResponseConsumer.onContentReceived(BasicAsyncResponseConsumer.java:82)
    at org.apache.http.nio.protocol.AbstractAsyncResponseConsumer.consumeContent(AbstractAsyncResponseConsumer.java:141)
    at org.apache.http.impl.nio.client.MainClientExec.consumeContent(MainClientExec.java:329)
    at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.consumeContent(DefaultClientExchangeHandlerImpl.java:153)
    at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:303)
    at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:267)
    at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
    at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
    at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:116)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:164)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:339)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:317)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:278)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:590)
    at java.lang.Thread.run(Thread.java:744)

I had set the

MaxDirectMemorySize=100m

, and i had set

-XX:-DisableExplicitGC.

And the main stack for poolingNhttpClientManager is following, it seems the select() stucks.

sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
   - locked sun.nio.ch.Util$2@3920a91
   - locked java.util.Collections$UnmodifiableSet@68f3b76b
   - locked sun.nio.ch.EPollSelectorImpl@588f07a1
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:342)
org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:191)
org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
java.lang.Thread.run(Thread.java:744)

score 3 · Answer 1 · edited May 23 '17 at 12:04

You're probably getting this because you're hitting the max limit for direct memory set by -XX:MaxDirectMemorySize. All the NIO DirectByteBuffer objects in your app like the one in the stack above are holding onto direct memory, which they don't release until they are garbage collected. As you can see in the OpenJDK code here, once the amount of direct memory these objects are holding exceeds MaxDirectMemorySize an OutOfMemoryError is thrown. Here's the relevant method:

static void reserveMemory(long size, int cap) {
    synchronized (Bits.class) {
        if (!memoryLimitSet && VM.isBooted()) {
            maxMemory = VM.maxDirectMemory();
            memoryLimitSet = true;
        }
        // -XX:MaxDirectMemorySize limits the total capacity rather than the
        // actual memory usage, which will differ when buffers are page
        // aligned.
        if (cap <= maxMemory - totalCapacity) {
            reservedMemory += size;
            totalCapacity += cap;
            count++;
            return;
        }
    }

    System.gc();
    try {
        Thread.sleep(100);
    } catch (InterruptedException x) {
        // Restore interrupt status
        Thread.currentThread().interrupt();
    }
    synchronized (Bits.class) {
        if (totalCapacity + cap > maxMemory)
            throw new OutOfMemoryError("Direct buffer memory");
        reservedMemory += size;
        totalCapacity += cap;
        count++;
    }

}

As you can also see in that method call, if there's insufficient direct memory available under the max, as a last ditch effort the jvm will explicitly invoke System.gc() to try to get some of the direct buffer objects to be GCed and hence also release their associated memory. So there are a few things you can do here:

Increase the value of -XX:MaxDirectMemorySize
Remove the -XX:-DisableExplicitGC setting which could allow some of those objects to be freed up in that last ditch call to System.gc() (although you generally want to avoid this for performance reasons)
Tweak your GC tuning so that the DirectByteBuffer objects will be collected through the normal GC process before you hit the max direct memory cap

You can get more information from a related post here: Impact of setting -XX:+DisableExplicitGC when NIO direct buffers are used .

httpasyncclient OutOfMemoryError Direct buffer memory

1 Answers1