Tomcat 6 memory leaks log entries

Question

Below is outtake of unique entries in my Catalina.out file on CentOS machine. I'm running Tomcat 6 with spring 3 and my application. There is whole bunch of them so I just picked some that keep repeating. This doesn't happen all the time but it happens at least once a week.

The Question is what can I do to prevent the bellow from happening?

Feb 3, 2011 2:37:48 PM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc

SEVERE: The web application [] registered the JBDC driver [com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.

Feb 3, 2011 2:37:48 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

SEVERE: The web application [] appears to have started a thread named [com.iteezy.shared.domain.DirEntry.data] but has failed to stop it. This is very likely to create a memory leak.


Feb 3, 2011 2:37:48 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [] appears to have started a thread named

[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0] but has failed to stop it. This is very likely to create a memory leak.


Feb 3, 2011 2:37:48 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads  

SEVERE: The web application [] appears to have started a thread named [File Reaper] but has failed to stop it. This is very likely to create a memory leak.

Feb 3, 2011 2:37:48 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

SEVERE: The web application [] appears to have started a thread named [pool-1-thread-22] but has failed to stop it. This is very likely to create a memory leak.  

37:48 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
b application [] appears to have started a thread named 

[org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-2] but has failed to stop it. This is very likely to create a memory leak.

37:48 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap

b application [] created a ThreadLocal with key of type [net.sf.json.AbstractJSON$1] (value [net.sf.json.AbstractJSON$1@40bbb3d6]) and a value of type [java.util.HashSet] (value [[]]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.

score 12 · Accepted Answer · edited May 23 '17 at 12:00

When you define an external flag which a thread is suppose to poll and exit when it's set - it must be volatile. Otherwise the thread might never see the change made by other thread.

However there is already a feature like that in standard API - it is called an interrupt() method and Thread.currentThread().isInterrupted(). No need to duplicate already existing logic. See: Stopping a specific java thread.

That being said calling interrupt() on each and every thread is a bad idea as well because there is no guarantee that all threads respond to it. Examining your exceptions I noticed the following threads not being cleaned up properly:

com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0 - close the C3P0 data source. Since you are using Spring simply add destroy-method="close". We're done with this thread.
File Reaper - as far as I can see this thread is created by FileCleaningTracker. You need to call FileCleaningTracker.exitWhenFinished() explicitly when shutting down your application (or when the class is no longer needed, I never used it) or let Spring to do this (see above). Chances are some 3rd party library uses it and not closing properly - this means it has a bug.
pool-1-thread-22 - this is one of the threads created by Executors utility inside ExecutorService. Make sure you call shutdown() on every such a pool in your application during shutdown.
org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-2 - Quartz worker thread (the one that actually runs jobs). SchedulerFactoryBean closes the scheduler for you automatically, I think Tomcat is mistaken here, I see this error often as well. Nevertheless looks like setting SchedulerFactoryBean.waitForJobsToCompleteOnShutdown to true solves this.
com.iteezy.shared.domain.DirEntry.data - I am not sure about this one. It is either your own thread that needs to be interrupted upon shutdown or H2 database thread (?) Its stack needs to be examined to guess where does it come from.

The bottom line is: don't just kill everything that moves (actually, Tomcat does that for you after issuing this warning) but determine where the threads come from and use framework/library specific close() method to allow further cleanup. Be gentle.

Rich · Answer 2 · 2011-02-07T15:15:40.473

1

Set up a Servlet to manage this in its destroy() method. The threads can check a flag to see if they must continue or not.

In your servlet, do the following in the destroy method. You will obviously need to be able to have access to a Collection<MyThread> but how you get that really depends on how your system is set up.

destroy() {
    for (MyThread thread : myThreads) {
        thread.stopProcessing();
    }
}

Your MyThread class will have something like this:

public class MyThread {
    private boolean finished = false;

    @Override
    public void run() {
        while (!finished) {
            //do something
        }
    }

    public void stopProcessing() {
        finished = true;
    }
}

edited Feb 07 '11 at 15:15

answered Feb 04 '11 at 14:56

Rich

15,602
15
79
126

Do I override the destroy method on each servlet on my system? Also, do I put the class MyThread under each servlet? – Bikash Gyawali Mar 10 '11 at 21:10
@Bikash No, only need to override destroy on one servlet, a servlet that is not actually called anywhere, but is initiated (web.xml). Then that servlet needs access to a Collection of threads, and it iterates through that collection in the destroy method. You obviously need to populate that collection when threads are created, and you need to make sure that threads that complete before Tomcat is shut down remove themselves from the collection otherwise you will get memory leaks! – Rich Mar 10 '11 at 21:53
`finished` has to be `volatile`, also why not simply use existing `interrupted` flag available for every thread? See: http://stackoverflow.com/questions/7786305 – Tomasz Nurkiewicz Oct 27 '11 at 08:09

score -1 · Answer 3 · answered Feb 04 '11 at 14:45

-1

Shutdown your webapp properly. Don't leave these threads running.

Alternatively, don't keep redeploying the webapp.

answered Feb 04 '11 at 14:45

OrangeDog

36,653
12
122
207

This is pretty unhelpful advice, akin to "just say no". The OP was shutting down the webapp properly to his mind, as a feature of Tomcat (and nearly every other servlet container) is to stop webapps without any special coding required. Tomascz's comment not only described what the error message was but also how to track it down. – Ed Griebel Aug 05 '13 at 16:22

Tomcat 6 memory leaks log entries

The Question is what can I do to prevent the bellow from happening?

3 Answers3

Linked