19

I have a Java application that is hosted on by a web hosting company. Every few days my app goes down with:

[2011-03-09 15:52:14,501] ERROR http-12021-9 
java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method)
    at java.lang.Thread.start(Thread.java:597)

The hosting company says it means my app is leaking memory, but the tools I have are showing free memory is still available. Since the error is always creating a new native thread my thinking is that the issue is in the JVM config/OS resources.

How do I prevent this error from happening?

Eric Leschinski
  • 146,994
  • 96
  • 417
  • 335
tvfoodmaps
  • 347
  • 1
  • 3
  • 14

7 Answers7

12

One possibility is that you have reached your user limit for the number of open files.

I believe that every Process/Thread consumes one or more file descriptors.

For example, when this occurs for your user then "no" shell command will work, since shell commands fork off a process to execute (you see errors like "-bash: fork: retry: Resource temporarily unavailable")

I hit this issue and found that only the current user was unable to spawn procs... other users were uneffected.

To resolve, up your ulimit -n (max files open) setting... details follow.

You can see your user limits with the command: 

ulimit -a

Up your max file limit with the following: 

ulimit -n 65536

Here is what I have right now:

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 256797
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 75000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 100000
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

To see all the explicit limits for your system:

cat /etc/security/limits.conf

Please note: I'm using Oracle Linux 6.3 - results may differ slightly between distros.

jatal
  • 790
  • 1
  • 10
  • 19
  • I also had trouble with Eclipse Luna on Fedora 20. I've read a lot about different memory options and still couldn't figure out the problem. It drove me crazy. Finally I realized that by default, the number of processes a regular user can have on Fedora is very limited. Content of limits.d/90-nproc.conf: * soft nproc 1000 Raising this to 5000 fixed my "Unable to create new native thread" problems. – arpadf Oct 08 '14 at 09:37
7

Its most likely the problem with JVM at web-server end. Please check out the following link for some details,

http://blog.egilh.com/2006/06/2811aspx.html

Tayyab
  • 9,951
  • 1
  • 16
  • 19
  • 5
    it could also be an issue with the application just creating way too many threads -- as the article mentions, more threads == more native space memory usage which can be addressed with tuning.. this assumes legitimate number of threads needed for an app. but if the app spawns threads and never kills them (thread leak) then no amount of tuning will fix it. it could also be that the OS has a limit on the thread count per-process. – Helter Scelter Mar 09 '11 at 23:18
  • Is there any easy way from java code itself to get the # of threads associated with my process? Since this is a webhosting company JVM I have very little access to JVM level tools. I can instrument my code though.. (which is what I did to get JVM memory info). – tvfoodmaps Mar 09 '11 at 23:33
  • there is no way AFAIK, but hosting company has support team who can dump the threads for you if you teach them how. – Vladimir Dyuzhev Mar 10 '11 at 02:35
2

When you fire up your process, the JVM has a limited heap size (default is 128MB). That server may well have more memory, but your JVM doesn't - you used it all.

You can change this with the -Xms and -Xmx command line arguments, but I would suggest finding the memory leak first :)

Brian Roach
  • 76,169
  • 12
  • 136
  • 161
  • 3
    I don't believe I would get the "unable to create new native thread" part of the OOM message if this was a standard memory leak. – tvfoodmaps Mar 09 '11 at 23:35
  • Understand I'm using that term in the most generic sense since there's no way to allocate memory directly in java then lose track of it like you could in C. How are you cleaning up these threads you start? – Brian Roach Mar 10 '11 at 03:20
  • Thank you, had this error with arduino and using Xms100m solved it! – Ramast Jan 09 '18 at 20:45
2

This answer is to all, who are running java via systemd. (e.g. self created tomcat service)

I was running my java application on the server via Tomcat. I also created a service unit in systemd for convenience, so it starts on server booting up and i can also control it via systemd (or service tomcat restart).

But there are some default values for systemd units and their maximum allowed tasks (threads). For me on my machine it was 195 Tasks. Once i changed the value in the service unit with TasksMax=1024 and reloaded it with systemctl daemon-reload everything worked as expected.

For example the service unit file in /etc/systemd/system/tomcat.service

[Unit]
Description=Tomcat9
After=network.target

[Service]
Type=forking
User=tomcat9
Group=tomcat9

TasksMax=1048

Environment=CATALINA_PID=/opt/tomcat/tomcat9.pid
Environment=JAVA_HOME=/usr/lib/jvm/default-java
Environment=CATALINA_HOME=/opt/tomcat
Environment=CATALINA_BASE=/opt/tomcat
Environment="CATALINA_OPTS=-Xms2048m -Xmx28384m"
Environment="JAVA_OPTS=-Dfile.encoding=UTF-8 -Dnet.sf.ehcache.skipUpdateCheck=true -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC"

ExecStart=/opt/tomcat/bin/startup.sh
ExecStop=/opt/tomcat/bin/shutdown.sh

[Install]
WantedBy=multi-user.target
Sunchezz
  • 740
  • 6
  • 21
1

Did you do any memory tracking? Fire up jconsole and watch or log your memory consumption over a 24 hour period. If it (on average) goes up without coming back down, then you are running out of memory and possibly have insufficient memory to store the details of your new thread.

Edwin Buck
  • 69,361
  • 7
  • 100
  • 138
1

It is the problem with linux to handle no.of open files Give as follows ulimit -n 65536 (any number u can give)

sugan
  • 11
  • 1
0

Looks like thread leaking. Threads are created but then stuck somewhere. Dump the threads periodically to see if the number of allocated threads is growing. Look for any sleeping/hanging threads in the dump.

kill -QUIT jvm_pid
Vladimir Dyuzhev
  • 18,130
  • 10
  • 48
  • 62