0

I have Apache Tomcat running on an PROD EC2 instance managed by Elastic Beanstalk.

Every five to six hours, incoming requests would fail with the error

java.lang.OutOfMemoryError: Unable to create new native thread

I initially thought that raising the JVM Heap Size in the EB configuration from the default of 256m to 1024m would help to alleviate the issue, but it still recurs.

Naturally, as my UAT environment does not have the load factor of the PROD environment, this issue did not crop up in UAT.

As the EC2 instances are sitting behind a bastion host, I'm not able to directly connect to it with VisualVM or other memory profiling tools.

Can anyone suggest how I should go about finding out what is causing this and resolving it once and for all?

Earthling
  • 283
  • 5
  • 17
  • It might be a thread leak. Try using [VisualVM](https://visualvm.java.net/) or a debugger to track the number of threads. – kichik Apr 28 '17 at 00:48
  • @kichik As mentioned in my earlier post, the EC2 instance is behind a bastion host, and my infra team will not allow me to connect to it with VisualVM. – Earthling Apr 28 '17 at 00:54
  • 1
    Then manually [dump them](http://stackoverflow.com/questions/1323408/get-a-list-of-all-threads-currently-running-in-java) and upload somewhere. – kichik Apr 28 '17 at 00:55
  • a `kill -3 java?pid` will dump the threads to stdout - of course stdout maybe going to a log file – Scary Wombat Apr 28 '17 at 01:06
  • @kichik All right; I've gotten the thread dump, and loaded it into VisualVM, but where do I go from here? – Earthling Apr 28 '17 at 01:07
  • @ScaryWombat Yup, managed to get that and loaded into VisualVM, however, me being new to all this, I'm suddenly clueless on how to proceed. – Earthling Apr 28 '17 at 01:09
  • Get one when the process starts and then another every so often. Check out if the number of threads keeps on growing. Try to figure out what creates those threads. – kichik Apr 28 '17 at 01:09
  • @kichik Meaning that I'll have to monitor my servers for it to happen again, then periodically do thread dumps? – Earthling Apr 28 '17 at 01:17
  • You have to do the thread dumps before it crashes, otherwise you won't have the proper data. – kichik Apr 28 '17 at 01:19
  • @kichik Yikes. That will be a tough one to catch. I'll see what I can do. Is there anything else I can do to try and trace the root cause? – Earthling Apr 28 '17 at 01:53
  • Get as much data as you can over time to improve your chances of finding the issue. – kichik Apr 28 '17 at 01:56

0 Answers0