1

We are migrating our java 8 application to java 17, and switching the GC from G1GC to ZGC. Our application is running as a container and the only difference between base-images of these two is the version of java. For example for java 17 version:

FROM ubuntu:20.04
...
ENV JAVA_HOME /usr/lib/jvm/jdk-17.0.5
ENV PATH $JAVA_HOME/bin:$PATH

And its related java options:

-Xmx100g
-XX:+UseZGC

When running application, these gc warnings appear:

[0.018s][warning][gc] ***** WARNING! INCORRECT SYSTEM CONFIGURATION DETECTED! *****
[0.018s][warning][gc] The system limit on number of memory mappings per process might be too low for the given
[0.018s][warning][gc] max Java heap size (102400M). Please adjust /proc/sys/vm/max_map_count to allow for at
[0.018s][warning][gc] least 184320 mappings (current limit is 65530). Continuing execution with the current
[0.018s][warning][gc] limit could lead to a premature OutOfMemoryError being thrown, due to failure to map memory.

And after a few hours, after memory reached about ~60GB, application crashed:

Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(...) failed; error='Not enough space' (errno=12)
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(...) failed; error='Not enough space' (errno=12)
# fatal error: Failed to map memory (Not enough space)
# A fatal error has been detected by the Java Runtime Environment:
[error occurred during error reporting (), id 0xb, SIGSEGV (0xb) at pc=0x00007fc929746941]

But our java 8 version of application had never encountered such a problem before. I know adjusting vm.max_map_count will most likely fix the problem', but my question is what exactly has been changed in java or GC which causes these warnings and eventually these errors?

Arya
  • 2,809
  • 5
  • 34
  • 56
  • Could you try any `G1GC` with JDK-17 and confirm your observation? Asking, because this seems to be a one-off case observed with one of our enterprise applications during migration. – Naman Mar 12 '23 at 15:39
  • You could also try setting enabling large pages or disabling uncommit, that might result in more coarse-grained memory mappings, but no guarantees, I haven't tried it. – the8472 Mar 12 '23 at 16:53
  • Why do you explicitly tell the JVM which garbage collector to use? – user18619318 Mar 16 '23 at 12:42
  • @user18619318 Because the default GC of java 17 is `G1GC`, But we want to use `ZGC`. – Arya Mar 16 '23 at 15:16
  • In that case I would strongly suggest _first_ getting it to work with the default configuration, so no "-X" flags what so ever, and _then_ come back to switching the garbage collector if actually needed. – user18619318 Mar 20 '23 at 08:38
  • Also you should fix what the error message tells you to fix to avoid the situation that the error message tells you might happen. – user18619318 Mar 20 '23 at 09:49

1 Answers1

2

but my question is what exactly has been changed in java or GC which causes these warnings and eventually these errors?

It's simply an implementation detail of the ZGC garbage collector. It needs more fine-grained control over its memory mappings which means a larger total amount of mappings.

The older collectors usually reserved huge contiguous spaces of address space and only rarely punched some holes into those ranges to release memory back to the OS. ZGC works with non-contiguous ranges of varying size which means it has to request more individual mappings from the OS.

the8472
  • 40,999
  • 5
  • 70
  • 122
  • Although ZGC is good choice for large heap size, isn't it strange that this is not mentioned in the official documentation? – Arya Mar 13 '23 at 18:04