Java+docker RSS tuning for glibc bug

Question

I have similar story as in question Growing resident memory usage (RSS) of Java Process

jcmd NMT shows no issues. It has constant values. For example

Native Memory Tracking:
Total: reserved=1611305KB +19680KB, committed=347717KB +41440KB

At the same time command docker exec eb97c7681baa ps -o rss,vsz,sz 7 returns

  RSS    VSZ    SZ
458368 2294800 573700

And the committed size for NMT remains +/- constant, but RSS is keep growing, which causes docker OOM killer to stop the container with OOM. I read several great articles about memory fragmentation ( https://www.ibm.com/developerworks/community/blogs/kevgrig/entry/linux_native_memory_fragmentation_and_process_size_growth?lang=en)

So a lot of people report that fix like MALLOC_ARENA_MAX=2 should resolve the issue, but it seems that more tuning is required. When I set the value the issue with growth of RSS decreases significantly, but still was in place. It seems that I need to setup values for:

MALLOC_MMAP_THRESHOLD_ - defines when memory will be allocated out of java heap (default value 128k)

MALLOC_TRIM_THRESHOLD_ it tries to trim arenas to release memory back to operating system when the arena size reaches max threshold (default value 128k) - Does it?

From the article:

If you decide to change the mmap threshold, the first step is to determine the allocation pattern. This can be done through tools such as ltrace (on malloc) or SystemTap, or if you know what is causing most of the allocations (e.g. Java DirectByteBuffers), then you can trace just those allocations. Next, create a histogram of these sizes and choose a threshold just under the smallest yet most frequent allocation. For example, let's say you've found that most allocations are larger than 8KB. In this case, you can set the threshold to 8192

I used pmap tool to check what are the sizes of allocation:

7:   java -XX:NativeMemoryTracking=detail -XX:+PrintFlagsFinal -XshowSettings:vm -jar /usr/share/jvmservice/service.jar
Address           Kbytes     RSS   Dirty Mode  Mapping
00000000020af000     560     432     432 rw---   [ anon ]
00000000f8000000   43648   39032   39032 rw---   [ anon ]
00000000faaa0000   87424   66532   66532 rw---   [ anon ]
0000000100000000   11136   11056   11056 rw---   [ anon ]
00007f2820000000   30912   25520   25520 rw---   [ anon ]
00007f2828000000   65512   58568   58568 rw---   [ anon ]
00007f282fb06000    1280    1176    1176 rw---   [ anon ]
00007f282fd06000    2048    2044    2044 rw---   [ anon ]
00007f282ff06000    2048    2048    2048 rw---   [ anon ]
00007f2830106000    2048    2048    2048 rw---   [ anon ]
00007f2830309000    3064    2064    2064 rw---   [ anon ]
00007f283060a000    1016     100     100 rw---   [ anon ]
00007f283070b000    3064    2068    2068 rw---   [ anon ]
00007f2830a09000    2048    2048    2048 rw---   [ anon ]
00007f2830c0c000    1016     108     108 rw---   [ anon ]
00007f2830d0d000    1016      92      92 rw---   [ anon ]
00007f2830e0e000    1016     124     124 rw---   [ anon ]
00007f2830f0f000    1016     128     128 rw---   [ anon ]
00007f2831111000    1016     124     124 rw---   [ anon ]
00007f2831212000    1016     128     128 rw---   [ anon ]
00007f2831313000    1016     144     144 rw---   [ anon ]
00007f2831414000    1016     128     128 rw---   [ anon ]
00007f2831515000    1016     144     144 rw---   [ anon ]
00007f2831616000    1016     128     128 rw---   [ anon ]
00007f2831717000    1016     136     136 rw---   [ anon ]
00007f2831818000    3064    2136    2136 rw---   [ anon ]
00007f2831b16000    2048    2044    2044 rw---   [ anon ]
00007f2831d19000    3064    2152    2152 rw---   [ anon ]
00007f2832017000    2048    2048    2048 rw---   [ anon ]
00007f283221a000    3064    2152    2152 rw---   [ anon ]
00007f2832518000   16388     164     164 rw---   [ anon ]
00007f283351c000    1016     136     136 rw---   [ anon ]
00007f283361d000    3064    2136    2136 rw---   [ anon ]
00007f283391b000    2048    2048    2048 rw---   [ anon ]
00007f2833d20000    1016     100     100 rw---   [ anon ]
00007f2833e21000    3064    2148    2148 rw---   [ anon ]
00007f283411f000    2048    2048    2048 rw---   [ anon ]
00007f2834b43000    3064    2136    2136 rw---   [ anon ]
00007f2835291000    2116    2052    2052 rw---   [ anon ]
00007f28354a2000    2048    2048    2048 rw---   [ anon ]
00007f28356a2000    2048    2048    2048 rw---   [ anon ]
00007f28358a2000    2048    2048    2048 rw---   [ anon ]
00007f2835aa2000    2048    2048    2048 rw---   [ anon ]
00007f2835ca2000    2048    2048    2048 rw---   [ anon ]
00007f2835ea2000    2048    2048    2048 rw---   [ anon ]
00007f28360e1000    2048    2048    2048 rw---   [ anon ]
00007f28365df000    3064    2164    2164 rw---   [ anon ]
00007f28368dd000    2048    2048    2048 rw---   [ anon ]
00007f2836add000    2048    2048    2048 rw---   [ anon ]
00007f2836cdd000    2048    2048    2048 rw---   [ anon ]
00007f2836ee0000    3064    2156    2156 rw---   [ anon ]
00007f28371de000    2048    2048    2048 rw---   [ anon ]
00007f28373de000    2048    2048    2048 rw---   [ anon ]
00007f28375de000    2048    2048    2048 rw---   [ anon ]
00007f2837c00000    2048    2048    2048 rw---   [ anon ]
00007f2837e00000    2048    2048    2048 rw---   [ anon ]
00007f2838000000   19828   19820   19820 rw---   [ anon ]
00007f283c000000    6296    6036    6036 rw---   [ anon ]
00007f2840000000    1004     952     952 rw---   [ anon ]
00007f2844000000   65520   62180   62180 rw---   [ anon ]
00007f2848000000     192     192     192 rw---   [ anon ]
00007f284c000000    3260    3228    3228 rw---   [ anon ]
00007f2850000000    5060    5060    5060 rw---   [ anon ]
00007f28544b7000    1024      12      12 rw---   [ anon ]
00007f28545ba000    1016     116     116 rw---   [ anon ]
00007f28546bc000    1012      24      24 rw---   [ anon ]
00007f28547bd000    1012      56      56 rw---   [ anon ]
00007f28548bd000    1016      92      92 rw---   [ anon ]
00007f2854b31000    1016      92      92 rw---   [ anon ]
00007f2854c32000    1016      88      88 rw---   [ anon ]
00007f2854d31000    1024      16      16 rw---   [ anon ]
00007f2855000000   45440   44788   44788 rwx--   [ anon ]
00007f2864000000   27756   26120   26120 rw---   [ anon ]
00007f28680d1000    8976    8976    8976 rw---   [ anon ]
00007f2868995000     712     712     712 rw---   [ anon ]

00007f286a0f7000   12612    4916       0 r-x-- libjvm.so
---------------- ------- ------- ------- 
total kB         2294804  461600  456396

So it seems that most cells are of 2048K size. Also I analysed heap with JXRay

It seems that this space is used by java DirectByteBuffers. Here is a great article how to tune it https://www.evanjones.ca/java-bytebuffer-leak.html

Questions 1. Will it be correct to set

MALLOC_MMAP_THRESHOLD_ = 8192

Question 2. Which consideration I need to take into account to set value for MALLOC_TRIM_THRESHOLD_. From docs

   M_TRIM_THRESHOLD
          When the amount of contiguous free memory at the top of the
          heap grows sufficiently large, free(3) employs sbrk(2) to
          release this memory back to the system.  (This can be useful
          in programs that continue to execute for a long period after
          freeing a significant amount of memory.)  The M_TRIM_THRESHOLD
          parameter specifies the minimum size (in bytes) that this
          block of memory must reach before sbrk(2) is used to trim the
          heap.

          The default value for this parameter is 128*1024.  Setting
          M_TRIM_THRESHOLD to -1 disables trimming completely.

          Modifying M_TRIM_THRESHOLD is a trade-off between increasing
          the number of system calls (when the parameter is set low) and
          wasting unused memory at the top of the heap (when the
          parameter is set high).

score 0 · Answer 1 · answered Jul 06 '19 at 10:54

The list of anonymous memory you posted does not look like something the glibc allocated would produce in a situation where there is excessive RSS consumption. You would see a lot more mappings with 64 MiB alignment (not necessarily of that size exactly, depending on the reporting tool, but the starting address would be a multiple of 0x4000000, that is, at least six trailing zeros. Therefore, it looks unlikely that glibc malloc tuning will help you address this issue.

You might have a genuine memory leak in your application. If it is indeed caused by direct byte buffers, you might want to try to lower the -XX:MaxDirectMemorySize to get an idea what the minimum amount of direct byte buffers is that your application needs. Another possibility is that your garbage collection settings happen to postpone direct byte buffer cleanup to such a degree that this causes the out-of-memory situation.

Thanks for reply. Currently I run the services on jvm12 with next settings. -jmvOptions="XX:+AlwaysPreTouch -Xmx150m -Xms150m -XX:+UseStringDeduplication" -DMALLOC_ARENA_MAX=2 -DMALLOC_MMAP_THRESHOLD_=8192 -Djdk.nio.maxCachedBufferSize=262144. Not sure if some of the setting helped? But the problem with unbounded memory allocation disappered — user2105282, Jul 18 '19 at 11:42
Environment variables need to be set using the shell, not the `-D` argument. — Florian Weimer, Jul 18 '19 at 17:33

Java+docker RSS tuning for glibc bug

1 Answers1