0

I am using Netty 4.1.17-Final.

I wrote the code to send and receive 100 MB random ASCII. Decoder does not read until ByteBuf becomes 100 MB.

@Override
public void decode(ByteBuffer _in, List<Object> _out) {
    if (_in.remaining() == size) {   // size = 100 * 1024 * 1024
        _in.get(new byte[size]);
    }
}

Therefore, Netty buffers 100 MB, but it was not found even by monitoring Direct Memory.

System.out.println(sun.misc.SharedSecrets.getJavaNioAccess().getDirectBufferPool().getMemoryUsed());
ManagementFactory.getPlatformMXBeans(BufferPoolMXBean.class).forEach(buf -> {
    System.out.printf("%s: Used=%d, TotalCap=%d%n", buf.getName(), buf.getMemoryUsed(), buf.getTotalCapacity());
});

// Result
// 10
// direct: Used=9, TotalCap=8
// mapped: Used=0, TotalCap=0

Netty used PooledUnsafeDirectByteBuf, but where do you buffer the received data? Or is the direct buffer monitoring method incorrect?

nikai
  • 27
  • 1
  • 6

2 Answers2

2

You can tell Netty to not use Unsafe for the direct buffers and so have it show up in JMX again. Unfortunally this also will have some performance impact.

Just use -Dio.netty.maxDirectMemory=0.

See: https://github.com/netty/netty/blob/4.1/common/src/main/java/io/netty/util/internal/PlatformDependent.java#L153

Norman Maurer
  • 23,104
  • 2
  • 33
  • 31
  • Thank you for your reply. My code now uses DirectBuffer. I found "io.netty.noUnsafe" and "io.netty.noPreferDirect". What is the difference between these and "io.netty.maxDirectMemory = 0"? – nikai Jun 21 '18 at 03:12
  • `..noUnsafe` disable the usage of `sun.misc.Unsafe` completely (which comes with a performance overhead in general). `...noPreferDirect` will have `allocator.buffer()` return an heap buffer and not a direct buffer. This may result in more memory copies. `...maxDirectMemory=0` will just ensure we use `DirectByteBuffer` under the cover and so JMX will work. So I would advice to use what I suggested if you have no other strong reason for not doing so. – Norman Maurer Jun 21 '18 at 06:23
  • Thank you politely. I will use `maxDirectMemory = 0` as you say. – nikai Jun 22 '18 at 01:01
0

Self-answer, In my code Netty had acquired DirectMemory using com.sun.Unsafe. This area can not be captured with BufferPoolMXBean or SharedSecrets.

I tried to get memory with Unsafe, but it was not output to the monitoring result.

Field unsafeField = Unsafe.class.getDeclaredField("theUnsafe");
unsafeField.setAccessible(true);
Unsafe unsafe = (Unsafe) unsafeField.get(null);
unsafe.allocateMemory(1000);

I found the following request.
extension for tracking off-heap memory use for netty · Issue #475 · Netflix/spectator

Please tell me if there is a way to get the size obtained by Unsafe.

nikai
  • 27
  • 1
  • 6