JVM memory is growing beyond limit in ECS container

Question

My task definition is configured with these limits:

"cpu": "1024",
"memory": "8192"

I'm running the jar inside a docker container using the "docker" cgroup flags:

java -XX:+UseContainerSupport -XX:MaxRAMPercentage=80 -XX:InitialRAMPercentage=70 /myjar.jar foo.Main

But ECS is killing my service with OOM errors.

I've gone ahead and measured the JVM memory usage and report it from within my application as a debug measure using the following:

val bean: MemoryMXBean = ManagementFactory.getMemoryMXBean
val hmu: MemoryUsage = bean.getHeapMemoryUsage
val nhu = bean.getNonHeapMemoryUsage
... reporting these metrics ...

In the image , the top portion is what cloud-watch is reporting as the used memory. As you can see it is at 100%.

The bottom graph shows the application reported memory:

val pc = (1.0 * hmu.getUsed) / hmu.getCommitted

From the documentation:

 * Below is a picture showing an example of a memory pool:
 *
 * <pre>
 *        +----------------------------------------------+
 *        +////////////////           |                  +
 *        +////////////////           |                  +
 *        +----------------------------------------------+
 *
 *        |--------|
 *           init
 *        |---------------|
 *               used
 *        |---------------------------|
 *                  committed
 *        |----------------------------------------------|
 *                            max
    /**
     * Returns the amount of memory in bytes that is committed for
     * the Java virtual machine to use.  This amount of memory is
     * guaranteed for the Java virtual machine to use.
     *
     * @return the amount of committed memory in bytes.
     *
     */
    public long getCommitted() {
        return committed;
    };

/**
     * Returns the amount of used memory in bytes.
     *
     * @return the amount of used memory in bytes.
     *
     */
    public long getUsed() {
        return used;
    };

My Docker file is very simple:

FROM openjdk:10-jdk

COPY service.jar /affinity-service.jar
COPY start.sh /start.sh
RUN chmod +x /start.sh
CMD ["/start.sh"]

and start.sh is:

#!/bin/bash
set -x

OPTS=""

#... setting flags from ENV values...
#...
#...

java -XX:+UseContainerSupport -XX:MaxRAMPercentage=80 -XX:InitialRAMPercentage=70 ${OPTS} -jar /service.jar com.....Service

Please elaborate further, I fail to understand what your question is. — Michael, Apr 01 '19 at 18:56
why is ECS memory usage growing to 100% while the JVM is reporting much less. The JVM memory graphs are very consistant and stable for several hours. Also, the allotted memory using the `-XX:+UseContainerSupport -XX:MaxRAMPercentage=80 -XX:InitialRAMPercentage=70` seems to exceed the memory allowed by the container's limits — Avba, Apr 01 '19 at 18:57
I suspect `MaxRAMPercentage` can't read the container limit and get the host RAM see. https://ops.tips/blog/why-top-inside-container-wrong-memory/ — Tilo, Dec 02 '20 at 04:56
the help claims they can read it: https://www.eclipse.org/openj9/docs/xxusecontainersupport/ — Tilo, Dec 02 '20 at 04:59

score 6 · Answer 1 · answered Apr 01 '19 at 21:02

6

MaxRAMPercentage and InitialRAMPercentage flags do not limit the memory of a Java process.
The only thing these flags affect is the heap size - see this answer for details.

As I've explained here, Java process can use much more memory than the heap size.

The bad news is - it's impossible to set the hard memory limit with just JVM flags to guarantee that JVM is never killed by the OS. But the same answer may give an idea how to analyze memory footprint of a Java process.

answered Apr 01 '19 at 21:02

apangin

92,924
10
193
247

what are the recommended flags to add when running an application server within a docker contianer which obeyes the high level restrictions? – Avba Apr 02 '19 at 11:41
@AvnerBarr Recommended flags are those that aren't set, i.e. have the default value. All other should be used to address particular problems. If you want to see why Java process consumes too much memory, I'd suggest to start with [Native Memory Tracking](https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr007.html). – apangin Apr 02 '19 at 20:00

JVM memory is growing beyond limit in ECS container

1 Answers1