0

I'm trying to understand why I'm getting out of memory errors in my cloud run application when cloud run never shows my memory allocation > 36% (purple line belo).

java.lang.OutOfMemoryError: Java heap space

enter image description here

Teal line is CPU usage and the purple is allocated memory. This scenario seems like we're CPU bound, but during this time period we are logging the java.lang.OutOfMemoryError errors. I'm pretty baffled. When exceeding max heap I'm used to seeing CPU usage rise during garbage collection and then a steep drop off in memory usage once garbage collection completes. The graph seems like we do not have a memory consumption issue because the memory allocation is staying pretty flat at 36%.

How can be we be blowing up the heap size if we are only reporting 36% of available memory allocated?

cobolstinks
  • 6,801
  • 16
  • 68
  • 97
  • You could get it if your code attempted to allocate something very large. Also don't confuse OS allocation with Java allocation. The VM could allocate 36% of your available RAM, but that doesn't mean that the heap is using 36% of RAM, it only means Java has allocated it for use as a heap. Viceversa, the heap could be using the fully allocated 36% thus running out of ram. – AminM Apr 03 '23 at 15:48
  • What are the min and max heap settings you are using and what garbage collector are you using? Best to share your full Java GC configuration options. Also, I'd suggest you use a profile or dump a log of GC events so you can see what is really going on with the Java heap. OS tools won't be enough. – AminM Apr 03 '23 at 15:48
  • hmm we're not setting an explicit max heap size setting anywhere. Our max container memory capacity is set to 4Gb. I guess I assumed that Cloud Run would manage how high it lets the heap grow based on the max container size? – cobolstinks Apr 03 '23 at 15:52
  • 1
    I'm not familiar with how GCP would control Java's heap size, but I'd look into your assumptions to make sure they're correct. Having it dump GC logs should clarify that as well. – AminM Apr 03 '23 at 15:54
  • Try this https://stackoverflow.com/questions/4667483/how-is-the-default-max-java-heap-size-determined To determine what the max heap size is in your environment. In one place I saw, it said max heapsize is 256 MB. Its important to look at your particular VM settings to ensure memory is being used correctly. – AminM Jun 16 '23 at 23:19

0 Answers0