0

Related to these two questions:

I run a Java application on Cloud Foundry and need to make sure that the allocated memory is not exceeded. Otherwise, and this is the current issue, the process is killed by Cloud Foundry monitoring mechanisms (Linux CGROUP).

The Java Buildpack automatically sets sane values for -Xmx and -Xss. By tuning the arguments and configuring the (maximum) number of expected threads, I'm pretty sure that the memory consumed by the Java process should be less than the upper limit which I assigned to my Cloud Foundry application.

However, I still experience Cloud Foundry "out of memory" errors (NOT the Java OOM error!): index: 3, reason: CRASHED, exit_description: out of memory, exit_status: 255

I experimented with the MALLOC_ARENA_MAX setting. Setting the value to 1 or 2 leads to slow startups. With MALLOC_ARENA_MAX=4 I still saw an error as described above, so this is no solution for my problem.

Currently I test with very tight memory settings so that the problem is easier to reproduce. However, even with this, I have to wait about 20-25 minutes for the error to occur.

Which arguments and/or environment variables do I have to specify to ensure that my Java process never exceeds a certain memory limit? Crashing with a Java OOM Error is acceptable if the application actually needs more memory.

Further information regarding MALLOC_ARENA_MAX:

EDIT: A possible explaination is this: http://www.evanjones.ca/java-bytebuffer-leak.html. As I currently see the OOM issue when doing lots of outgoing HTTP/REST requests, these buffers might be to blame.

Community
  • 1
  • 1
C-Otto
  • 5,615
  • 3
  • 29
  • 62
  • How tight is tight? We've seen that the JVM `native` memory consumption balloons on startup (it's pretty opaque as to what's going on) but eventually returns all that native memory to the OS as startup proceeds. You might be running into that. – Ben Hale Dec 09 '15 at 19:21
  • Tight as in less than 10 MByte of free space, I think I managed to get it to less than 1 MByte. And how long does that ballooning take? Seconds? Minutes? – C-Otto Dec 09 '15 at 20:33
  • Seconds, but typically long enough that it will trigger the container's defenses. If you're seeing the container get involved later than that, it's likely you're seeing `native` memory growth. – Ben Hale Dec 10 '15 at 21:17
  • OK. Thank you. My service crashes after about 20 minutes. – C-Otto Dec 11 '15 at 07:16
  • 1
    "Run services with -Djdk.nio.maxCachedBufferSize=262144 to avoid this problem." explained in Evan Jones's blog post is a useful tip. That feature was backported to 8u102, see https://bugs.openjdk.java.net/browse/JDK-8147468 – Lari Hotari Feb 15 '19 at 20:47

1 Answers1

2

Unfortunately, there is no way to definitively enforce a memory limit on the JVM. Most of the memory regions are configurable (-Xmx, -Xss, -XX:MaxPermSize, -XX: MaxMetaspaceSize, etc.) but the one you can't control is Native memory. Native memory contains a whole host of things from memory mapped files to native libraries to JNI code. The best you can do is profile your application, find out where the memory growth is occurring, and either solve the growth or give yourself enough breathing room to survive.

Certainly unsatisfying, but in the end not much different from other languages and runtimes that have no control over their memory footprint.

Ben Hale
  • 567
  • 3
  • 7