0

Am deploying springboot application in kubernets using Jib. When the service starting the memory usage is around 300MB but it grows up to 1.3gb over time. How to avoid this increase without any usage? The application is up and running. The API gateways are not open to user now still the memory is incrementing over time.

kubernets deployment configuration

# Source: services/charts/login/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/instance: release-name
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: login
    app.kubernetes.io/version: 1.16.0
  name: login
spec:
  selector:
    matchLabels:
      app.kubernetes.io/instance: release-name
      app.kubernetes.io/name: login
  template:
    metadata:
      labels:
        app.kubernetes.io/instance: release-name
        app.kubernetes.io/name: login
    spec:
      containers:
      - env:
        - name: APP_NAME
          value: login-release-name
        - name: JAVA_TOOL_OPTIONS
          value: -Dspring.profiles.active=prod
        image: dockerregistry.com/login:1.0.0
        imagePullPolicy: Always
        lifecycle:
          preStop:
            exec:
              command:
              - sh
              - -c
              - sleep 10
        livenessProbe:
          httpGet:
            path: /actuator/health
            port: 8080
          initialDelaySeconds: 30
        name: login
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
        readinessProbe:
          httpGet:
            path: /actuator/health
            port: 8080
          initialDelaySeconds: 30
        resources:
          limits:
            cpu: 2000m
            memory: 1Gi
          requests:
            cpu: 100m
            memory: 1Gi
      imagePullSecrets:
      - name: regcred
      terminationGracePeriodSeconds: 60

spring boot configuration for kubernets

server.port=8080
server.shutdown=graceful
spring.lifecycle.timeout-per-shutdown-phase=45s

server.tomcat.accept-count=100
server.tomcat.max-connections=8000
server.tomcat.connection-timeout=10000
server.tomcat.max-threads=200
server.tomcat.min-spare-threads=10

spring.datasource.url=jdbc:postgresql://${DB_HOST:#{"postgres"}}/postgres
spring.datasource.username=${DB_USER:#{"postgres"}}
spring.datasource.password=${DB_PASSWORD:#{"na"}}
spring.datasource.type=org.springframework.jdbc.datasource.DriverManagerDataSource
spring.datasource.driver-class-name=org.postgresql.Driver

Do we need to configure anything to limit the memory usage to 1GB limit? Now the kubernets will kill the pod if it goes beyond 1GB.

am creating the image using the Jib.

mvn compile com.google.cloud.tools:jib-maven-plugin:3.3.0:dockerBuild -Dimage=login -DskipTests
PRAJIN PRAKASH
  • 1,366
  • 1
  • 15
  • 31

2 Answers2

2

Update 05 02 2023 Changing the JVM Runtime to OpenJ9 reduced memory footprint dramatically,

also Changed to jre image instead of jdk

Updated command mvn compile com.google.cloud.tools:jib-maven-plugin:3.3.0:dockerBuild -Djib.from.image=ibm-semeru-runtimes:open-8-jre -Dimage=$imageName -DskipTests

OLD Answer

After a long analysis I found that there is no memory leak from the application. It was due to the container so I switched to openjdk:8-jdk-alpine using Djib.from.image= option in Jib

first I added some JVM option to set The JVM Heap But its also behaving same.

-XX:+UseG1GC -Xms100M -Xmx1G (Garbage collection changed and set max heap to 1GB

using VisualVM suggest by @Mihai Pasca I analyzed the memory usage in the container.

updated command of Jib

mvn compile com.google.cloud.tools:jib-maven-plugin:3.3.0:dockerBuild -Djib.from.image=openjdk:8-jdk-alpine -Dimage=$imageName -DskipTests  -Djib.container.environment=JAVA_TOOL_OPTIONS="-XX:+UseG1GC -Xms100M -Xmx1G -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=9010 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.rmi.port=9010 -Djava.rmi.server.hostname=localhost"  -Djib.container.ports=8080,9010 
 

And connected to 9010 from visual vm using add JMX option.

java memory usage in visual vm

There is no over usage I found and the GC works fine

And memory usage by the container also fine and not growing continuesly its gorwing up and come down if no use. Previusly its continuly growing.

docker desktop memory usage

now I started the same in k8s working fine now.

PRAJIN PRAKASH
  • 1,366
  • 1
  • 15
  • 31
1
  • To limit max heap size for your Spring Boot app add max memory argument to JAVA_TOOL_OPTIONS (-Xmx1024M)
  • To see why memory consumption grows, use VisualVM (https://visualvm.github.io/download.html) to connect to your process and take a heap dump and analyze it
  • This will only set Heap size but the memory use is above that. I alredy I set `- name: JAVA_TOOL_OPTIONS value: "-Dspring.profiles.active={{ .Values.activeProfile }} -Xmx768m -Xms768m"` but the memory usage reaches 970mb and the autoscaler will create new instances if it reaches 95% of memory – PRAJIN PRAKASH Jan 26 '23 at 14:47
  • 1
    There are a couple of things to investigate / tweak: a) are the java parameters being used correctly? b) what else is running in the container besides jvm and how much memory it uses? c) check this https://stackoverflow.com/questions/49854237/is-xxmaxramfraction-1-safe-for-production-in-a-containered-environment – Mihai Pasca Jan 26 '23 at 15:58