0

Using Netty Server Builder for gRPC Server & running into Out of Memory error where direct memory size is exceeding the allocated size of 1.2g.

Tried using both pooled byte buffer allocator as well as unpooled byte buffer allocator. Looks like Netty is creating Thread Local Cache and the size of the thread local cache is almost equal to the total size of direct memory. Tried setting the below system property to disable thread local cache but still see that the thread local cache is getting created.

Initially was using Netty version 4.1.74.FINAL but also upgraded Netty server version 4.1.89.Final and seeing the same issue

Any suggestions on how to use Netty Server effectively for creating & managing the gRPC server.

Tried using both pooled byte buffer allocator as well as unpooled byte buffer allocator. Looks like Netty is creating Thread Local Cache and the size of the thread local cache is almost equal to the total size of direct memory. Tried setting the below system property to disable thread local cache but still see that the thread local cache is getting created.

user2649857
  • 11
  • 1
  • 2

1 Answers1

0

I would suggest that you either use upooled or a small thread local cache:

  1. set io.netty.allocator.type="unpooled"
  2. set io.netty.threadLocalDirectBufferSize=16384

This is based on the default for threadLocalDirectBufferSize resulting in threadlocal buffers large enough to hold the entire message (which since you said that each are using up almost all of the memory must be large) and that the system is going to break large messages up into 16KB chunks for sending over the wire, so you are just moving where the chunking is occurring.

Original that generated comments:

You didn't list the command that you had tried. Was it

-Dio.netty.allocator.useCacheForAllThreads=false

That should only be relevant with PooledByteBufferAllocator.

If you are using UnpooledByteBufAllocator, then there shouldn't be any thread local caches.

I am assuming you are following standard practice and setting the allocator in ChannelConfig, is that correct?

Larry Safran
  • 109
  • 5
  • Using this system property System.setProperty("io.netty.noThreadLocalDirectBuffers", "true") - And am using io.grpc.NettyServerBuilder to build the gRPC server – user2649857 Mar 07 '23 at 00:33
  • Tried using System.setProperty("io.netty.allocator.useCacheForAllThreads", "false") but still I see that the thread local cache is created in the direct arena of the pooled byte buffer allocator – user2649857 Mar 07 '23 at 01:17
  • Looking at the code, if you don't specify io.netty.allocator.type= unpooled", then you'll get a PooledByteBufAllocator. If you don't specify io.netty.threadLocalDirectBufferSize, then that defaults to 0 which results in the buffer being allocated as the same size as the ByteBuffer being written. – Larry Safran Mar 08 '23 at 01:50