tomcat java.util.logging process

Question

I have a linux (ubuntu) server running tomcat7. The server has 1GB of ram, and I am getting heap space errors thrown from java.

Basically what is happening, is that my tomcat server (running my servlet code) is throwing exceptions because there is no more memory to allocate. My code is pretty lean, so I don't think that is the problem.

When I look at htop, I see a lot of java logging processes, I see about 30 of these processes:

ps ax | grep java
 7412 pts/0    Sl     0:02 /usr/bin/java -Djava.util.logging.config.file=/usr/share/tomcat/apache-tomcat-7.0.29/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/usr/share/tomcat/apache-tomcat-7.0.29/endorsed -classpath /usr/share/tomcat/apache-tomcat-7.0.29/bin/bootstrap.jar:/usr/share/tomcat/apache-tomcat-7.0.29/bin/tomcat-juli.jar -Dcatalina.base=/usr/share/tomcat/apache-tomcat-7.0.29 -Dcatalina.home=/usr/share/tomcat/apache-tomcat-7.0.29 -Djava.io.tmpdir=/usr/share/tomcat/apache-tomcat-7.0.29/temp org.apache.catalina.startup.Bootstrap start
 7426 pts/0    D+     0:00 grep --color=auto java
25937 ?        Sl    13:12 /usr/bin/java -Djava.util.logging.config.file=/usr/share/tomcat/apache-tomcat-7.0.29/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/usr/share/tomcat/apache-tomcat-7.0.29/endorsed -classpath /usr/share/tomcat/apache-tomcat-7.0.29/bin/bootstrap.jar:/usr/share/tomcat/apache-tomcat-7.0.29/bin/tomcat-juli.jar -Dcatalina.base=/usr/share/tomcat/apache-tomcat-7.0.29 -Dcatalina.home=/usr/share/tomcat/apache-tomcat-7.0.29 -Djava.io.tmpdir=/usr/share/tomcat/apache-tomcat-7.0.29/temp org.apache.catalina.startup.Bootstrap start

These java processes slowly leaks memory over time, and because I only have 1gb of ram to play with my tomcat server starts throwing heap space exceptions.

I have fiddled with log4j (but it should be turned off) and I have done all I can to turn off logging in configs I can find.

Any help would be great, I just need to get rid of these processes - they are way to greedy. Thanks!

~ Dan

EDIT: MORE INFO:

This is probably a big enough hint to solve this one entirely - I checked some of the startup debug and came up with this:

    Oct 02, 2012 8:50:08 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

SEVERE: The web application [/NMVServer] appears to have started a thread named [http-bio-8080-exec-53] but has failed to stop it. This is very likely to create a memory leak.
Oct 02, 2012 8:50:08 AM org.apache.catalina.loader.WebappClassLoader 
clearReferencesThreads

SEVERE: The web application [/NMVServer] appears to have started a thread named [http-bio-8080-exec-54] but has failed to stop it. This is very likely to create a memory leak.
Oct 02, 2012 8:50:08 AM org.apache.catalina.loader.WebappClassLoader 
clearReferencesThreads

SEVERE: The web application [/NMVServer] appears to have started a thread named [http-bio-8080-exec-62] but has failed to stop it. This is very likely to create a memory leak.
Oct 02, 2012 8:50:08 AM org.apache.catalina.loader.WebappClassLoader 
clearReferencesThreads

SEVERE: The web application [/NMVServer] appears to have started a thread named [http-bio-8080-exec-64] but has failed to stop it. This is very likely to create a memory leak.

One problem with leaks that I found is due to a bug with the JDBC Driver http://bugs.mysql.com/bug.php?id=36565 according to a couple of forums. That solves another message I was getting.

I have been looking around and I have found this which is quite useful:

http://wiki.apache.org/tomcat/MemoryLeakProtection

Now I am just trying to work through all your advise thanks heaps!

~ Dan

Please run `ps ax | grep java` and paste one *full* command line. The part you have pasted only shows a bunch of configuration options. The actual class being called is not visible... — thkala, Aug 17 '12 at 11:24
Sorry about the delay and the dodgey initial post. I have updated it with proper info. — ddoor, Aug 19 '12 at 23:03
You mean you have about 30 processes which are similar to one with pid 7412 in your post? But that process represents the whole Tomcat server, as can be seen from "org.apache.catalina.startup.Bootstrap start". That means you have 30 servers up and running? — Nikem, Oct 01 '12 at 13:00
sorry, that was a bit ambiguous, I think it was more htop's interpretation of the processes running, a {ps ax} was more accurate. Its only one process which is the bootstrapper for the JVM I think — ddoor, Oct 02 '12 at 00:03

score 3 · Accepted Answer · edited May 23 '17 at 11:49

3

I general I would expect the logging code in java to have been well tested. If there is no log output being generated by your logging config it has a low overhead. Your htop results are probably showing some idle thread status in tomcat.

It could be logging but you may want to consider all the ways your application or another component may be leaking memory.

All JVM's have pretty good tools for diagnosing memory leaks. Threads like this discuss great ways to find out what is going on.

Edit: some more thoughts

You might want to check your configuration for too many sessions if a lot of requests are using HttpServletRequest.getSession(). The Manager component config goes inside Context and defaults to -1 for unlimited sessions.

Check your version of tomcat with the current issue fix list in case you are encountering a known memory leak.

Lastly, Tomcat 7 includes a JreMemoryLeakPreventionListener that might show resources not being released.

edited May 23 '17 at 11:49

Community

1
1

answered Sep 24 '12 at 09:45

pd40

3,187
3
20
29

1

Thanks, that has given me a great starting point. I knew that there were a few ways to fool the garbage collector (so to speak) - I have started investigating my mysql-connector pool and looking at other areas. Thanks for the advise. – ddoor Sep 25 '12 at 00:18
P.S. Is there anything in particular regarding servelets and a heavy usage of the "doPost" routine? – ddoor Sep 25 '12 at 00:18
Generally doPost() are stateless and should not hold any connections or create any permanent objects. Heavy load should not matter. – pd40 Sep 25 '12 at 01:01

score 1 · Answer 2 · edited Nov 19 '12 at 21:13

Tomcat usually has a thread pool. The Java EE specification requires to start a separate thread for processing each HTTP request. Tomcat has a thread pool and reuses threads from this pool instead of creating and terminating them. More details are available here

You can set the max memory limit for Tomcat by modifying setenv.sh (or setenv.bat respectively) eg:

CATALINA_OPTS='-Xms512m -Xmx1024m'

More details for the Oracle JVM are available here

But in practice the most important thing is to track memory allocation inside the JVM. You will need Java VisualVM for this.

tomcat java.util.logging process

2 Answers2