What is the smallest maximum memory a simple Spring Boot container can work with without getting OOMKilled?

Question

I have a Spring boot app that talks to Mongo DB using spring-boot-starter-data-mongodb. I am using bootBuildImage to build the container. To make the app work under 180 MB, I tried to tweak as follows:

build.gradle

bootBuildImage {
    imageName = "gcr.io/kubegcp-256806/coderprabhu-api:${project.version}"
    environment = [
            "BPL_JVM_HEAD_ROOM" : "2",
            "BPL_JVM_LOADED_CLASS_COUNT" : "35",
            "BPL_JVM_THREAD_COUNT" : "10"
        ]
}

deployment.yaml

        resources:
          requests:
            cpu: "0.02"
            memory: "64Mi"
          limits:
            cpu: "0.3"  
            memory: "180Mi" 
        env:
        - name: SPRING_PROFILES_ACTIVE
          value: "prod"    
        - name: BPL_JVM_HEAD_ROOM
          value: "2"  
        - name: BPL_JVM_LOADED_CLASS_COUNT
          value: "35"  
        - name: BPL_JVM_THREAD_COUNT
          value: "10"  
        - name: JAVA_OPTS
          value: >-
                -XX:ReservedCodeCacheSize=80M
                -XX:MaxMetaspaceSize=60M
                -Xlog:gc
                -Xms110m
                -Xmx140m
                -XX:MaxRAM=180M
                -XX:+PrintFlagsFinal

Build pack that bootBuildImage uses has 240M for just ReservedCodeCache size. So tried to squeeze is down and spring boot with spring data mongo db seems to be ok with 80MB. I get OOMKilled if MaxMetaspaceSize is less than 60 MB. I only have 2 tomcat threads to serve here.

Final GC Flags

kubectl logs coderprabhu-api-app-pod | grep command

      int ActiveProcessorCount  = 1            {product} {command line}
   size_t InitialHeapSize       = 115343360    {product} {command line}
   size_t MaxHeapSize           = 146800640    {product} {command line}
   size_t MaxMetaspaceSize      = 62914560     {product} {command line}
 uint64_t MaxRAM                = 188743680 {pd product} {command line}
     bool PrintFlagsFinal       = true         {product} {command line}
    uintx ReservedCodeCacheSize = 83886080  {pd product} {command line}

The container gets OOMKilled after some idle time and I see following logs.

allocated memory is greater than 180M available for allocation: -XX:MaxDirectMemorySize=10M, -Xmx140M, -XX:MaxMetaspaceSize=60M, -XX:ReservedCodeCacheSize=80M, -Xss1M x 10 threads%                                                                                    

The node/express app on same server is happily serving complex functions at less than 30MB of total usage.

What is the smallest maximum memory a simple Spring Boot container can work with without getting OOMKilled?

Answer 1

Out of Memory error cause:

In the deployment yaml in question, I had JVM arg -XX:MaxRAM=180M and also container limit memory: "180Mi". This made JVM to think that it has access to all 180M and with no consideration of memory taken up by OS processes, causing out of memory errors. I increased container limit to memory: "190Mi" and reduced memory visible to JVM using -XX:MaxRAM=150M. 40MB buffer should be more than enough for occasional kubectl exec -its for bash terminal and top command and other OS overheads.

Further memory usage reduction:

For this simple CRUD app the default Thread stack size of 1MB was too much and I was able to lower it to 256k. -Xss256k

From VisualVM, app uses max meta space of 41MB so, I set max meta space of 60 MB. -XX:MaxMetaspaceSize=60M

Also Visual VM showed that after GC heap space comes down to 20 MB, so I set Max heap space to 40 MB to give some wiggle room. -Xmx40m

Update on Xms on 6/11: Xms of 30MB was causing higher startup time and eventually heap memory was increasing to 32 MB anyways. So I have updated to -Xms34m. Startup now completes in 27 seconds.

The app has been running with no out of memory errors for past 40 hours.

Kubernetes Stats:

watch 'kubectl get pods|grep coderprabhu-api & kubectl top pods|grep coderprabhu-api' coderprabhu-api-app-55b5774544-xgkxv 1/1 Running 0 20h coderprabhu-api-app-55b5774544-xgkxv 2m 164Mi

GC Logs:

Heap after GC invocations=182 (full 3): def new generation total 12288K, used 130K eden space 10944K, 0% used from space 1344K, 9% used to space 1344K, 0% used tenured generation total 27328K, used 20475K the space 27328K, 74% used Metaspace used 54241K, capacity 55807K, committed 55984K, reserved 102400K class space used 6472K, capacity 7135K, committed 7168K, reserved 53248K

I have been loading the app every once in a while with 500 requests and saw 90th percentile response of 115ms

ab -n 500 -c 2 https://api.coderprabhu.com/count

The pod has been using around 171 to 175MB of memory as seen on Google Cloud Dashboard.

Google Cloud Usage

Following two articles from Dave Syer have been useful:

Spring Boot Memory Performance

How Fast is Spring?

Current configuration: containers: - name: coderprabhu-api-app image: gcr.io/kubegcp-256806/coderprabhu-api:0.0.9-SNAPSHOT imagePullPolicy: Always ports: - containerPort: 8080 resources: requests: cpu: "0.02" memory: "64Mi" limits: cpu: "0.3" memory: "190Mi" env: - name: SPRING_PROFILES_ACTIVE value: "prod" - name: BPL_JVM_HEAD_ROOM value: "2" - name: BPL_JVM_LOADED_CLASS_COUNT value: "35" - name: BPL_JVM_THREAD_COUNT value: "10" - name: JAVA_OPTS value: >- -XX:ReservedCodeCacheSize=40M -XX:MaxMetaspaceSize=60M -Xlog:gc*=debug -Xms34m -Xmx40m -Xss256k -XX:MaxRAM=150M -XX:+PrintFlagsFinal

Answer 2

You cannot put down hard numbers on Java programs, as this is strongly dependent on the actual JVM in use, and what the code does.

Given what you know and what you want to do, the best approach is most likely to familiarize yourself with the characteristics of the JVM you want to use. When and how the garbage collector works, how much the memory pressure is, what happens when your application grows. Etc.

I would suggest using the VisualVM tool to do this.

https://visualvm.github.io/