Permgen Memory Leak with hot redeploy and Oracle database

Question

i've got a SpringRoo-based application that is running in a production environment, that is causing severe permgen memory leaks after some hot redeploys.

In order to "find and fix" the leak and to reduce the variables during analysis, i've created a simple, streamlined application using roo, and i obtain the same behaviour. The project (created with Spring Roo (1.2.3.RELEASE)) simply persists an entity called 'Person' with a string field called 'name'.

I deploy the war on a Tomcat 7.0.39, using Oracle 11.2.0.2 as the database. After every re-deploy, i get this message in catalina.out

INFO: Undeploying context [/ojdbc-0.1.0.BUILD-SNAPSHOT]
mag 06, 2013 10:50:43 AM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc
SEVERE: The web application [/ojdbc-0.1.0.BUILD-SNAPSHOT] registered the JDBC driver       [oracle.jdbc.OracleDriver] but failed to unregister it when the web application was stopped.    To prevent a memory leak, the JDBC Driver has been forcibly unregistered.

After two hot-redeploys, i get a permgen error:

mag 06, 2013 10:51:08 AM org.apache.catalina.startup.HostConfig deployWAR
INFO: Deploying web application archive /Applications/apache-tomcat-7.0.39/webapps/ojdbc-    0.1.0.BUILD-SNAPSHOT.war
Exception in thread "ContainerBackgroundProcessor[StandardEngine[Catalina]]"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "ContainerBackgroundProcessor[StandardEngine[Catalina]]"
Exception in thread "RMI TCP Connection(idle)" mag 06, 2013 10:51:17 AM ServerCommunicatorAdmin reqIncoming
WARNING: The server has decided to close this client connection.
java.lang.OutOfMemoryError: PermGen space
Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError: PermGen space

I've tried to analyze this with VisualVm / EclipseMemory Analyzer, and this is what i get so far.

The fact is that i don't observe this behaviour with another database (for example PostgreSQL or Hypersonic). Is there something related to Oracle that is causing the leak?

Here is a zip archive containing roo script generator, anche the .hprof dump file.

Answer 1

Try moving the Oracle JDBC driver into Tomcat's lib directory instead of being inside of the application's lib folder. It looks like OracleDiagnosabilityMBean is getting a handle on Catalina.

Edit: Since you do not have control over Tomcat, try to wrap where the Oracle classes are getting loaded like so (except replace the AppContext for your Oracle init):

http://cdivilly.wordpress.com/2012/04/23/permgen-memory-leak/

//somewhere in application startup, e.g. the ServletContextListener
try {
 final ClassLoader active = Thread.currentThread().getContextClassLoader();
 try {
  //Find the root classloader
  ClassLoader root = active;
  while (root.getParent() != null) {
   root = root.getParent();
  }
  //Temporarily make the root class loader the active class loader
  Thread.currentThread().setContextClassLoader(root);
  //Force the AppContext singleton to be created and initialized
  sun.awt.AppContext.getAppContext();
 } finally {
 //restore the class loader
 Thread.currentThread().setContextClassLoader(active);   
}
} catch ( Throwable t) {
   //Carry on if we get an error
   LOG.warning("Failed to address PermGen leak");
}

Answer 2

There is an internal Oracle bug report

Oracle Support Bug 18707079 (JDBC THIN DRIVER LEAKS THREADS AND REFERENCES ON WEBAPP SHUTDOWN): https://support.oracle.com/epmos/faces/BugDisplay?id=18707079

which describes your problem:

When the webapp is shutdown, customer experience the following problems:
1) The OracleDiagnosabilityMBean is not de-registered
2) OracleTimeoutPollingThread does not stop
3) The following thread does not stop: 
oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser

It is set to Fixed in Product Version 12.2. Hopefully it will be released soon!

Answer 3

Try this:

Register a ServletContextListener in web xml

<listener>
    <listener-class>my.package.MyShutdownServletContextListener</listener-class>
</listener>

source:

public class MyShutdownServletContextListener implements ServletContextListener {
@Override
public void contextInitialized(ServletContextEvent event) {
    final ClassLoader contextClassLoader = Thread.currentThread().getContextClassLoader();
    try {
        Thread.currentThread().setContextClassLoader(ClassLoader.getSystemClassLoader());
        java.sql.DriverManager.getDrivers();
        Class.forName("oracle.jdbc.driver.OracleTimeoutThreadPerVM");
    } catch (ClassNotFoundException e) {
        /* noop */
    } finally {
        Thread.currentThread().setContextClassLoader(contextClassLoader);
    }
}
@Override
public void contextDestroyed(ServletContextEvent event) {}
}