How to identify and remove Threads/ThreadLocals initiated from our webapp in Java?

Question

Whenever I stop or redeploy the webapp, I see lot of errors similar to,

msg=The web application [] created a ThreadLocal with key of type [] (value []) and 
a value of type [] (value []) but failed to remove it when the web application was 
stopped. Threads are going to be renewed over time to try and avoid probable memory leak

I'm not creating any ThreadLocals in my app but referencing many libraries which may be creating these ThreadLocals. We are currently using Tomcat 7. I've already gone through other similar questions [Memory leak when redeploying application in Tomcat or What are these warnings in catalina.out?] but all of them only suggest that this is Tomcat feature to warn you about ThreadLocals not being removed. I don't see any answer to remove ThreadLocals. I also see few ERRORs regarding thread not stopped as well,

msg=The web application [] appears to have started a thread named [] but has
failed to stop it. This is very likely to create a memory leak.

These are being logged in as ERRORs in our company's central logging system and thereby increasing the error count by our application. This certainly does not look good when we check the performance of our app. I tried the implementations from these two sources [Killing threads and Sample code from this thread], but doesn't seem to work. It removes thread/threadlocals not created by our app. What I need is to remove only the threads/threadlocals started by our webapp. Is there any way we can remove these in contextDestroyed() method of ServletContextListener? Following is my current ServletContextListener class,

public class CustomServletContextListener implements ServletContextListener {

private List<String> threadsAtStartup;

@Override
public void contextInitialized(ServletContextEvent sce) {
    retrieveThreadsOnStartup();
}

@Override
public void contextDestroyed(ServletContextEvent sce) {
    // Now deregister JDBC drivers in this context's ClassLoader:
    // Get the webapp's ClassLoader
    ClassLoader cl = Thread.currentThread().getContextClassLoader();
    // Loop through all drivers
    Enumeration<Driver> drivers = DriverManager.getDrivers();
    while (drivers.hasMoreElements()) {
        Driver driver = drivers.nextElement();
        if (driver.getClass().getClassLoader() == cl) {
            // This driver was registered by the webapp's ClassLoader, so deregister it:
            try {
                System.out.println("Deregistering JDBC driver {}: " + driver);
                DriverManager.deregisterDriver(driver);
            } catch (SQLException ex) {
                System.out.println("Error deregistering JDBC driver {}: " + driver + "\nException: " + ex);
            }
        } else {
            // driver was not registered by the webapp's ClassLoader and may be in use elsewhere
            System.out.println("Not deregistering JDBC driver {} as it does not belong to this webapp's ClassLoader: " + driver);
        }
    }

    //Threads
    ThreadGroup threadGroup = Thread.currentThread().getThreadGroup();
    threadGroup = Thread.currentThread().getThreadGroup();
    Thread[] threads;
    try {
        threads = retrieveCurrentActiveThreads(threadGroup);
    } catch (NoSuchFieldException e) {
        System.out.println("Could not retrieve initial Threads list. The application may be unstable on shutting down " + e.getMessage());
        return;
    } catch (IllegalAccessException e) {
        System.out.println("Could not retrieve initial Threads list. The application may be unstable on shutting down " + e.getMessage());
        return;
    }

    int toBeKilledCount = 0;
    int totalThreadCount = 0;
    int killedTLCount = 0;
    int totalTLCount = 0;
    int killedITLCount = 0;
    int totalITLCount = 0;
    for (; totalThreadCount < threads.length; totalThreadCount++) {
        Thread thread = threads[totalThreadCount];
        if(thread != null) {
            String threadName = thread.getName();
            boolean shouldThisThreadBeKilled;

            shouldThisThreadBeKilled = isThisThreadToBeKilled(Thread.currentThread(), thread);
            if (shouldThisThreadBeKilled) {
                //ThreadLocal
                try {
                    removeThreadLocals("threadLocals", thread);
                    removeThreadLocals("inheritableThreadLocals", thread);
                } catch (Exception e) {
                    System.out.println("\tError accessing threadLocals field of '" + threadName + "': " + e.getMessage());
                }

                //Stop thread
                thread.interrupt();
                thread = null;
                toBeKilledCount++;
            }

        }
    }
}

private void retrieveThreadsOnStartup() {
    final Thread[] threads;
    final ThreadGroup threadGroup = Thread.currentThread().getThreadGroup();
    try {
        threads = retrieveCurrentActiveThreads(threadGroup);
    } catch (NoSuchFieldException e) {
        System.out.println("Could not retrieve initial Threads list. The application may be unstable on shutting down " + e);
        threadsAtStartup = new ArrayList<String>();
        return;
    } catch (IllegalAccessException e) {
        System.out.println("Could not retrieve initial Threads list. The application may be unstable on shutting down " + e);
        threadsAtStartup = new ArrayList<String>();
        return;
    }

    threadsAtStartup = new ArrayList<String>(threads.length);
    for (int i = 0; i < threads.length; i++) {
        final Thread thread;
        try {
            thread = threads[i];
            if (null != thread) {
                threadsAtStartup.add(thread.getName());
            }
        } catch (RuntimeException e) {
            System.out.println("An error occured on initial Thread statement: " + e);
        }
    }
}

private Thread[] retrieveCurrentActiveThreads(ThreadGroup threadGroup) throws NoSuchFieldException, IllegalAccessException {
    final Thread[] threads;
    final Field privateThreadsField;
    privateThreadsField = ThreadGroup.class.getDeclaredField("childrenThreads");
    privateThreadsField.setAccessible(true);

    threads = (Thread[]) privateThreadsField.get(threadGroup);
    return threads;
}

private void removeThreadLocals(String fieldName, Thread thread) {
    Field threadLocalsField = Thread.class.getDeclaredField(fieldName);
    threadLocalsField.setAccessible(true);
    Object threadLocalMap = threadLocalsField.get(thread);
    Field tableField = threadLocalMap.getClass().getDeclaredField("table");
    tableField.setAccessible(true);
    Object table = tableField.get(threadLocalMap);
    int count = 0;
    for (int i = 0, length = Array.getLength(table); i < length; ++i) {
        Object entry = Array.get(table, i);
        if (entry != null) {
            totalTLCount++;
            Object threadLocal = ((WeakReference)entry).get();
            if (threadLocal != null) {
                Array.set(table, i, null);
                killedTLCount++;
            }
        } 
    }   
}

private Boolean isThisThreadToBeKilled(Thread currentThread, Thread testThread) {
    boolean toBeKilled;
    String currentThreadName = currentThread.getName();
    String testThreadName = testThread.getName();
    System.out.println("currentThreadName: " + currentThreadName + ", testThreadName: " + testThreadName);
    return !threadsAtStartup.contains(testThreadName)               // this thread was not already running at startup
            && !testThreadName.equalsIgnoreCase(currentThreadName); // this is not the currently running thread

}

}

Update: I'm still not able to resolve this. Any help? Nobody ever ran into these?

Answer 1

There is no solution to fix all threadlocal leaks in one go. Normally third party libraries using Threadlocal variables have some kind of cleanup API call which can be used to clear their local thread variables.

You have to check for all reported threadlocal leaks and find the correct way of disposing them in the corresponding library. You can do this in your CustomServletContextListener

examples:

log4j (javadoc):

LogManager.shutdown()

jdbc driver: (javadoc):

DriverManager.deregisterDriver(driver);

note: Also check for new versions of your 3rd party libs to check fox fixes concerning memory leaks (and/or thread local leaks).

Answer 2

Solution depends on the library that created these Threads/ThreadLocal-s. Basically you need to call library's cleanup code from your CustomServletContextListener.contextDestroyed() method.

So find what is the library and how to shut it down properly.

Answer 3

You can try this code to remove all ThreadLocal

private void cleanThreadLocals() {
    try {
        // Get a reference to the thread locals table of the current thread
        Thread thread = Thread.currentThread();
        Field threadLocalsField = Thread.class.getDeclaredField("threadLocals");
        threadLocalsField.setAccessible(true);
        Object threadLocalTable = threadLocalsField.get(thread);

        // Get a reference to the array holding the thread local variables inside the
        // ThreadLocalMap of the current thread
        Class threadLocalMapClass = Class.forName("java.lang.ThreadLocal$ThreadLocalMap");
        Field tableField = threadLocalMapClass.getDeclaredField("table");
        tableField.setAccessible(true);
        Object table = tableField.get(threadLocalTable);

        // The key to the ThreadLocalMap is a WeakReference object. The referent field of this object
        // is a reference to the actual ThreadLocal variable
        Field referentField = Reference.class.getDeclaredField("referent");
        referentField.setAccessible(true);

        for (int i=0; i < Array.getLength(table); i++) {
            // Each entry in the table array of ThreadLocalMap is an Entry object
            // representing the thread local reference and its value
            Object entry = Array.get(table, i);
            if (entry != null) {
                // Get a reference to the thread local object and remove it from the table
                ThreadLocal threadLocal = (ThreadLocal)referentField.get(entry);
                threadLocal.remove();
            }
        }
    } catch(Exception e) {
        // We will tolerate an exception here and just log it
        throw new IllegalStateException(e);
    }
}

Answer 4

If we remove objects from thread local of all threads while stopping the container then we are only trying to solve the problem of error messages which are displayed by container. Rather, the objective should be to avoid memory leaks which may occurr over the period of time when container is not stopped/restarted. So, ideally as threads from ThreadPool are reused to serve different requests then after response is sent then there should not be any reason to hold memory by keeping objects in thread local because that thread may be used to serve next (totally different) request from client. One of the suggestion is to remove any objects from thread local by configuring Filter which is executed immediately before response is sent from server.

Answer 5

I would try to find which library causes these ThreadLocal, possibly by running the web app in a debugger and stopping on ThreadLocal creation. Then you can see if you forgot to clean up behind some library or if a library is buggy/not made for web app use. Maybe post your findings here.

When cleaning threads in a context listener, I once did check that the contextClassLoader of the thread was the same as the thread running the listener, to avoid messing the threads of other applications.

Hope this helps.

Answer 6

try

Runtime.getRuntime().addShutdownHook(webapp.new ShutdownHook());

in your shudownhook, clean the objects

Answer 7

If you use ThreadLocal in your code you could replace that ThreadLocal with ImrpovedThreadLocal that I made and you will not have a memory leak on stop/redeploy. You can use that threadLocal in the same way without having any thread contention.