0

I am running a java app hosted on a tomcat server inside a docker container using AWS EC2 (ECS). Recently the java app service is getting restarted after some time. After checking the ECS agent log it is showing OutOfMemoryError.

Logs:

Level: INFO
Message: DockerGoClient: Process within container 02118442aba47dbe536651b90ca1cd64c43ec419ea949dbe8ca3797f0df1dd71 (name: "my-java-app10-f8e5abadedfbd0c9d901") died due to OOM
Module: docker_client.go

Level: INFO
Message: Handling container change event
Task: 3825f1c6-0d3d-4771-bed7-15332dab4ad9
Container: service-layer
RuntimeID: 02118442aba47dbe536651b90ca1cd64c43ec419ea949dbe8ca3797f0df1dd71
Status: STOPPED

Level: WARN
Message: Error stopping the container; marking it as stopped anyway
Task: 3825f1c6-0d3d-4771-bed7-15332dab4ad9
Container: service-layer
RuntimeID: 02118442aba47dbe536651b90ca1cd64c43ec419ea949dbe8ca3797f0df1dd71
ErrorName: OutOfMemoryError
Error: Container killed due to memory usage

I am not able to figure out which java thread/process is consuming memory. I tried below:

docker inspect <containerName>
docker ps --all (This shows Exited 137))

Checked application logs as well where I could not see any specific calls.

Can anyone give me clue how should I check which java thread/process is running constantly and consuming memory. Note that the max memory limit is 1GB and it was working fine earlier with this.

tamtejszystan
  • 148
  • 7
Raghav91
  • 1
  • 1
  • 2
    There are tools to inspect running Java applications and find out such information. From IDE debuggers to Java agents to tools like VisualVM. – Jorn Aug 10 '23 at 07:21
  • If you started noticing the problem recently, it can be due to many reasons such as new code introduced causes higher usage of heap, update of dependencies, etc. This [post](https://stackoverflow.com/questions/53451103/java-using-much-more-memory-than-heap-size-or-size-correctly-docker-memory-limi/53624438#53624438) might help to understand what consumes memory in a Java process. You might also be interested in [Amazon CodeGuru Profiler](https://aws.amazon.com/blogs/devops/understanding-memory-usage-in-your-java-application-with-amazon-codeguru-profiler/) for profiling live applications. – SwathiP Aug 10 '23 at 09:40

1 Answers1

0

This is a java debugging issue. I can't help with that, but I can help with the crashing and restarting of the task.

In the meantime, you can adjust the Task Definition for your ECS Task to have a larger Task Size for CPU/Memory so that it stabilizes while you figure out why the service is consuming more memory.

You might consider using Fargate as the capacity provider instead of launching on EC2, so that you don't have conflicts with the Task Size in the Task Definition and the size of the EC2 instances associated with your cluster. You should know that there's cost implications (in case you're trying to keep it on EC2 free tier).

Steven Evers
  • 16,649
  • 19
  • 79
  • 126