14

In a Kubernetes cluster with numerous microservices, one of them is used exclusively for a Java Virtual Machine (JVM) that runs a Java 1.8 data processing application.

Up to recently, jobs running in that JVM pod consumed less than 1 GB of RAM, so the pod has been setup with 4 GB of maximum memory, without any explicit heap size settings for the JVM.

Some new data now require about 2.5 GB for the entire pod, including the JVM (as reported by the kubernetes top command, after launching with an increased memory limit of 8 GB), but the pod crashes soon after starting with a limit of 4 GB.

Using a head size range like -Xms256m -Xmx3072m with a limit of 4 GB does not solve the problem. In fact, now the pod does not even start.

Is there any way to parameterize the JVM for accommodating the 2.5 GB needed, without increasing the 4 GB maximum memory for the pod?

trincot
  • 317,000
  • 35
  • 244
  • 286
PNS
  • 19,295
  • 32
  • 96
  • 143
  • 2
    What version of the JVM? Current versions know to look for the pod's resource limits and size the heap accordingly, without any `-Xmx` options. – David Maze Oct 18 '20 at 00:04
  • Java 1.8 (updated in the question). – PNS Oct 18 '20 at 00:49
  • More specifically? Java 8u191 in particular is [capable of figuring it out on its own](https://www.oracle.com/technetwork/java/javase/8u191-relnotes-5032181.html#JDK-8146115). – David Maze Oct 18 '20 at 03:04
  • The question, how it's currently worded, has no sense. If a process takes 2.5 GB RAM, there will be no problem with running it in a container having 4 GB memory limit. When talking about 2.5 GB, do you mean Java Heap size, process resident set size (RSS), or virtual memory size (VSZ)? [This answer](https://stackoverflow.com/a/53624438/3448419) may give you an idea what makes the memory footprint of a Java process. – apangin Oct 18 '20 at 13:53
  • The total RAM consumption for the pod reaches 2.5 GB (question updated with more details). Based on that, a limit of 4 GB should have been enough, but it is not and an increase to 8 GB is necessary for the pod to run and the memory measurement to be taken, via the kubernetes top command. – PNS Oct 18 '20 at 23:05
  • Again, if a pod takes 2.5 GB RAM in total, it won't be killed because of a 4 GB limit. Look at the reason of a crash - there must be a message from OOM killer in kernel logs, or some clue in Kubernetes logs. – apangin Oct 19 '20 at 09:04

2 Answers2

26

The default "max heap" if you don't specify -Xmx is 1/4 (25%) of the host RAM. JDK 10 improved support for containers in that it uses container's RAM limits instead of underlying host. As pointed by @David Maze this has been backported to JDK 8.

Assuming you have a sufficiently recent version of JDK 8, you can use -XX:MaxRAMPercentage to modify the default percentage of total RAM used for Max heap. So instead of specifying -Xmx you can tell, e.g. -XX:MaxRAMPercentage=75.0. See also https://blog.arkey.fr/2020/10/27/maxrampercentage-is-not-what-i-wished-for/

Here's an example using alpine JDK docker image: https://hub.docker.com/_/openjdk (see section "Make JVM respect CPU and RAM limits" in particular).

# this is running on the host with 2 GB RAM
docker run --mount type=bind,source="$(pwd)",target=/pwd -it openjdk:8

# running with MaxRAMPercentage=50 => half of the available RAM is used as "max heap"
root@c9b0b4d9e85b:/# java -XX:+PrintFlagsFinal -XX:MaxRAMPercentage=50.0 -version | grep -i maxheap
    uintx MaxHeapFreeRatio                          = 100                                 {manageable}
    uintx MaxHeapSize                              := 1044381696                          {product}
openjdk version "1.8.0_265"
OpenJDK Runtime Environment (build 1.8.0_265-b01)
OpenJDK 64-Bit Server VM (build 25.265-b01, mixed mode)

# running without MaxRAMPercentage => default 25% of RAM is used
root@c9b0b4d9e85b:/# java -XX:+PrintFlagsFinal -version | grep -i maxheap
    uintx MaxHeapFreeRatio                          = 100                                 {manageable}
    uintx MaxHeapSize                              := 522190848                           {product}
openjdk version "1.8.0_265"
Juraj Martinka
  • 3,991
  • 2
  • 23
  • 25
  • We have already tried of all these facts and configuration options, but none of them works. As a matter of fact, we are running OpenJDK 1.8 versions newer than 8u200, but somehow the option `-XX:MaxRAMPercentage` is still not recognized. – PNS Oct 18 '20 at 22:57
  • 1
    At least the latest alpine JDK docker image seems to support it - see my edited answer and also https://hub.docker.com/_/openjdk – Juraj Martinka Oct 19 '20 at 03:02
  • 1
    How did you check that it's "not recognized"? Did you use `-XX:+PrintFlagsFinal`, some monitoring tool or it's just a guess depending based on the behavior of your application? – Juraj Martinka Oct 19 '20 at 03:03
  • 1
    @JurajMartinka When you claim that the JVM uses the `container's RAM limits instead of underlying host`, are you referencing `spec.containers[].resources.limits.memory` specifically? Wouldn't that cause a problem if you were to for example set a limit of 2Gb and a request of 1Gb, and only get access to 1Gb? You would be 1 GB short of the limit... – Copy and Paste Jul 08 '22 at 19:08
  • Yes, In kubernetes it should relate to `.resources.limits.memory` - discussed in more detail here: https://blog.arkey.fr/2020/10/27/maxrampercentage-is-not-what-i-wished-for/ – Juraj Martinka Jul 20 '22 at 12:05
  • 1
    Blog link seems to have been relocated to https://www.atamanroman.dev/development/2019/09/11/usecontainersupport-to-the-rescue.html – Jaap Apr 25 '23 at 14:41
0

In my K8s setup, I am using consul to manage the pod configuration. Here is a command to override the jvm setting on the fly. It is a pretty much project specific but it might give you a hint if you are using consul for configuration.

kubectl -n <namespace> exec -it consul-server -- bash -c "export CONSUL_HTTP_ADDR=https://localhost:8500 && /opt/../home/bin/bootstrap-config --token-file /opt/../config/etc/SecurityCertificateFramework/tokens/consul/default/management.token kv write config/processFlow/jvm/java_option_xmx -Xmx8192m"
Pankaj Shinde
  • 107
  • 1
  • 5