0

I have Java application, wrapped into Docker and hosted on Amazon ECS.

Main technologies are:

  • SpringBoot 2
  • Java 8
  • EC2 VM with 4Gb RAM

Every 1-2 days the application is crashing and waking up again resurrected by amazon service.

Under Docker insect I found the reason:

"OOMKilled": true,

I have actuator connected to CloudWatch and I found strange behaviour with memory consumption plot:

Memory consumption

Those blue peaks are the moments where the application has crashed.

I've read that Java 8 has some problem with reading correctly max memory for host but:

1) Docker is starting with flags that should solves the problem

ENTRYPOINT exec java -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=1 -Xms300M -XX:PermSize100M -Djava.security.egd=file:/dev/./urandom -jar /app.jar

2) Anyway VM has 4Gb ram, so ~1.3Gb peaks high, shouldn't kill the application.

3) There wasn't any significant load on server in the moments of peaks

Maciej Rudnicki
  • 131
  • 2
  • 7
  • Those blue peaks are moments when the applications has run out of memory; you haven't posted any code, but I would **guess** you have a memory leak. – Elliott Frisch May 18 '19 at 13:46
  • Yes, Is obvious that OOM goes because app run out of memory in these moment. I cannot upload here entire application. I'm looking for any Breadcrumps why it's happened and what to look for – Maciej Rudnicki May 18 '19 at 14:01

1 Answers1

1

-XX:PermSize100M

Not applicable for java 8.

-XX:MaxRAMFraction=1

This tells the JVM that the maximum of the managed heap for java objects is all of the available memory. But the JVM to allocate more memory than just for the java objects. E.g. metaspace, byte buffers for open files, loaded native libraries and so on.

You should do one of the following instead:

  • use MaxRAMFraction=2 instead of 1
  • manually set Xmx to something lower than the available memory
  • Upgrade to java >= 11 and use MaxRAMPercentage for more finegrained control
  • enable swap to let the additional memory spill to disk

Also see "Is -XX:MaxRAMFraction=1 safe for production in a containered environment?" for additional information regarding newer java versions.

the8472
  • 40,999
  • 5
  • 70
  • 122
  • sound reasonable. I'll try your point's. Maybe except switch to java 8. I'm not sure if all used libraries will be compatible. If app won't have any crashes for 3 days, I would consider problem as solved – Maciej Rudnicki May 18 '19 at 14:40
  • Ok, followed by your advices I found the article https://www.javaadvent.com/2018/12/docker-and-the-jvm.html and: there is state that if I won't set explicitly xmx, it is set automatically to 1GB. I checked and it's true. VM has 4 GB, but Docker add a limit to 1 GB. Java has a peeks and goes above 1GB, and application goes restart – Maciej Rudnicki May 18 '19 at 15:03