10

I have tomcat memory leak issue when stop/redeploy application. It says The following web applications were stopped (reloaded, undeployed), but their classes from previous runs are still loaded in memory, thus causing a memory leak (use a profiler to confirm):/test-1.0-SNAPSHOT

MySQL connector driver located in Tomcat/lib folder. I can reproduce this issue in both: Tomcat 7/8. Also tried MS SQL database with "net.sourceforge.jtds.*" driver but didn't help.

Please find below project files. Project only creates 1 table in DB.

build.gradle

group 'com.test'
version '1.0-SNAPSHOT'

apply plugin: 'java'
apply plugin: 'war'
sourceCompatibility = 1.8
repositories {
    mavenCentral()
}
dependencies {
    compile group: 'org.hibernate', name: 'hibernate-entitymanager', version: '5.2.10.Final'
    compile group: 'org.springframework.data', name: 'spring-data-jpa', version: '1.11.4.RELEASE'
    compile group: 'org.springframework', name: 'spring-webmvc', version: '4.3.9.RELEASE'
    providedCompile 'javax.servlet:javax.servlet-api:3.1.0'
    providedCompile group: 'mysql', name: 'mysql-connector-java', version: '5.1.6'
    compile group: 'commons-dbcp', name: 'commons-dbcp', version: '1.4'
}

ApplicationConfig.java

@Configuration
@Import({JPAConfiguration.class})
@EnableWebMvc
public class ApplicationConfig {}

JPAConfiguration.java

@Configuration
@EnableJpaRepositories("com.test.dao")
@EnableTransactionManagement
public class JPAConfiguration {

    @Bean
    public EntityManagerFactory entityManagerFactory() {
        LocalContainerEntityManagerFactoryBean factory = new LocalContainerEntityManagerFactoryBean();
        factory.setJpaVendorAdapter(new HibernateJpaVendorAdapter());
        factory.setPackagesToScan("com.test.model");
        factory.setDataSource(restDataSource());
        factory.setJpaPropertyMap(getPropertyMap());
        factory.afterPropertiesSet();
        return factory.getObject();
    }

    @Bean(destroyMethod = "close")
    public DataSource restDataSource() {
        BasicDataSource dataSource = new BasicDataSource();
        dataSource.setDriverClassName("com.mysql.jdbc.Driver");
        dataSource.setUrl("jdbc:mysql://localhost:3306/test");
        dataSource.setUsername("test");
        dataSource.setPassword("test");
        return dataSource;
    }

    private Map<String, String> getPropertyMap() {
        Map<String, String> hibernateProperties = new HashMap<>();
        hibernateProperties.put("hibernate.hbm2ddl.auto", "update");
        hibernateProperties.put("hibernate.show_sql", "true");
        hibernateProperties.put("hibernate.format_sql", "true");
        hibernateProperties.put("hibernate.dialect", "org.hibernate.dialect.MySQL5InnoDBDialect");
        return hibernateProperties;
    }

    @Bean
    public PlatformTransactionManager transactionManager() {
        JpaTransactionManager txManager = new JpaTransactionManager();
        txManager.setEntityManagerFactory(entityManagerFactory());
        return txManager;
    }

}

TestRepository.java

@Repository
public interface TestRepository extends JpaRepository<TestEntity, Long> {}

TestEntity.java

@Entity
@Table(name = "ent")
public class TestEntity {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;
    private String descript;
    //equals, hashcode, toString, getters, setters
}

AppInitializer.java

public class AppInitializer extends AbstractAnnotationConfigDispatcherServletInitializer {
    private WebApplicationContext rootContext;

    @Override
    protected Class<?>[] getRootConfigClasses() {
        return new Class[]{ApplicationConfig.class};
    }

    @Override
    protected Class<?>[] getServletConfigClasses() {
        return null;
    }

    @Override
    protected String[] getServletMappings() {
        return new String[]{"/"};
    }

}

Command

jmap -histo <tomcat_pid>

shows only 2 items from project structure after tomcat stop:

com.test.config.dao.JPAConfiguration$$EnhancerBySpringCGLIB$$792cb231$$FastClassBySpringCGLIB$$45ff499c
com.test.config.dao.JPAConfiguration$$FastClassBySpringCGLIB$$10104c1e

Anyone have ideas or suggestions to fix this problem?

Eugene Gotovko
  • 103
  • 1
  • 7
  • This link might be helpful https://stackoverflow.com/questions/40040289/java-spring-application-has-memory-leak-system-non-heap-memory-increases-consta – hrdkisback Jul 12 '17 at 10:49
  • For starters stop overriding `onStartup` and `createRootApplicationContext` you are creating an additional context which is pretty much useless and remove the field as well. You are trying to be to smart and with that are messing with the lifecycle. I would also suggest to remove the `setDriverClassName` line that isn't needed as JDBC is pretty much capable of figuring out the driver based on the URL. – M. Deinum Jul 12 '17 at 10:51
  • Thanks for responses. @M.Deinum, I removed `onStartup` and `createRootApplicationContext` methods but didn't help. Tomcat still shows message about memory leak. Tried to remove `setDriverClassName` but have an exception `org.hibernate.engine.jdbc.spi.SqlExceptionHelper.logExceptions Cannot create JDBC driver of class "" for connect URL "jdbc:mysql://localhost:3306/test"` – Eugene Gotovko Jul 12 '17 at 12:36
  • GitHub repo with sources https://github.com/egotovko/tomcat-leak – Eugene Gotovko Aug 11 '17 at 14:11
  • Given the above git, I see no memory leak when stop tomcat in my local machine. Could you elaborate more? What action did you do? – Mạnh Quyết Nguyễn Jun 19 '18 at 17:05
  • @MạnhQuyếtNguyễn, 1) make a war from the source provided. 2) then in Tomcat manager http://localhost:8080/manager/html deploy this war file. 3) then click "Undeploy" for deployed app 4) Click "Find leaks" in Tomcat manager. You'll see the message I provided in description. After these steps you can see classes from this application in thread dump for tomcat process. – Eugene Gotovko Jun 19 '18 at 17:15
  • Ya in my local machine there's no log like yours – Mạnh Quyết Nguyễn Jun 19 '18 at 17:17

2 Answers2

6

There are 2 memory leaks in this small project:

  1. The problem with MySQL jdbc driver.

We have to add ContextLoaderListener to deregister jdbc driver:

Listener:

@WebListener
public class ContextListener extends ContextLoaderListener {

    private final Logger log = LoggerFactory.getLogger(this.getClass());

    @Override
    public void contextInitialized(ServletContextEvent sce) {
        log.info("-= Context started =-");

    }

    @Override
    public void contextDestroyed(ServletContextEvent sce) {
        super.contextDestroyed(sce);
        log.info("-= Context destroyed =-");
        try {
            log.info("Calling MySQL AbandonedConnectionCleanupThread checkedShutdown");
            com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.uncheckedShutdown();

        } catch (Exception e) {
            log.error("Error calling MySQL AbandonedConnectionCleanupThread checkedShutdown {}", e);
        }

        ClassLoader cl = Thread.currentThread().getContextClassLoader();

        Enumeration<Driver> drivers = DriverManager.getDrivers();
        while (drivers.hasMoreElements()) {
            Driver driver = drivers.nextElement();

            if (driver.getClass().getClassLoader() == cl) {

                try {
                    log.info("Deregistering JDBC driver {}", driver);
                    DriverManager.deregisterDriver(driver);

                } catch (SQLException ex) {
                    log.error("Error deregistering JDBC driver {}", driver, ex);
                }

            } else {
                log.info("Not deregistering JDBC driver {} as it does not belong to this webapp's ClassLoader", driver);
            }
        }
    }
}

or if you had an access to tomcat server you can modify listener in tomcat/conf/server.xml example.

  1. The second problem is known memory leak in jboss-logging library (link).

The memory leak has gone after we exclude this library from hibernate dependency:

build.gradle:

group 'com.test'
version '1.0-SNAPSHOT'

apply plugin: 'java'
apply plugin: 'war'
sourceCompatibility = 1.8
repositories {
    mavenCentral()
}
dependencies {
    compile(group: 'org.hibernate', name: 'hibernate-entitymanager', version: '5.2.10.Final') {
        exclude group: 'org.jboss.logging', module: 'jboss-logging'
    }

    compile group: 'org.springframework.data', name: 'spring-data-jpa', version: '1.11.4.RELEASE'

    compile group: 'org.springframework', name: 'spring-webmvc', version: '4.3.9.RELEASE'

    providedCompile 'javax.servlet:javax.servlet-api:3.1.0'
    providedCompile group: 'mysql', name: 'mysql-connector-java', version: '8.0.11'
    compile group: 'org.slf4j', name: 'slf4j-api', version: '1.7.25'
    compile group: 'org.slf4j', name: 'slf4j-simple', version: '1.7.25'
}

then build jar from repo and added to tomcat /lib folder.

The issue with jboss-logging probably fixed in Java 9 (pull request link).

Andrew Nepogoda
  • 1,825
  • 17
  • 24
5

Short answer - hopefully the same problem for you...

Those two com.test.config.dao.JPAConfiguration$$...CGLIB$$... classes were being referenced indirectly by the Abandoned connection cleanup thread in MySQL:

20-Jun-2018 21:25:22.987 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [test-1.0-SNAPSHOT] appears to have started a thread named [Abandoned connection cleanup thread] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.lang.Object.wait(Native Method)
 java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
 com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:43)

The following answer enabled me to resolve the problem. E.g. in tomcat/conf/server.xml, look for the JreMemoryLeakPreventionListener line and replace it with this:

<Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" 
    classesToInitialize="com.mysql.jdbc.Driver" />

This forces the MySQL JDBC driver, and its cleanup thread, to be loaded outside the classloader for the web application. This means the cleanup thread won't hold a reference to the webapp classloader as its context class loader.


Expanded answer - how to trace the leak in your environment...

Hopefully the above is all you need - it was enough to reproduce and solve the problem against https://github.com/egotovko/tomcat-leak

However there are many other causes of a leaked reference to a web application that can stop it undeploying. E.g. other threads still running (Tomcat is good at warning about these) or references from outside the web application.

To properly trace the cause, you can chase the reference in a heap dump. If this is not familiar, you can get a heap dump from jmap -dump:file=dump.hprof <pid>, or by directly connecting from such as jvisualvm (also included in the JDK).

With the heap dump open in jvisualvm:

  • Select the Classes button for the heap dump
  • Sort the list of classes by name
  • Look for classes in the web application - e.g. com.test.config.dao.JPAConfiguration$$EnhancerBySpringCGLIB$$ in this example
  • This should be showing with an instance count of 2 or so
  • Double click to show these in the Instances View
  • In the References pane for one of these instances, right click and Show Nearest GC Root
  • E.g. for that Abandoned connection cleanup thread in MySQL: Referencing thread in JVisualVM

Note how the AbandonedConnectionCleanupThread has a contextClassLoader, which is the ParallelWebappClassLoader for the web application. Tomcat needs to be able to release the class loader to undeploy the web application.

Once you've tracked down what's holding the reference, it's then normally a case of investigating how better to configure that library in Tomcat, or perhaps someone else has seen that memory leak. It's also not uncommon to have to repeat the exercise, when there are several references to clear up.

df778899
  • 10,703
  • 1
  • 24
  • 36
  • Thank you. Changes with `JreMemoryLeakPreventionListener` fixes the problem with `AbandonedConnectionCleanupThread` and allow us to keep `mysql-connector-java` dependency in "provided" scope. As mentioned in the next answer there is second issue with leaks in this application related to `org.jboss.logging` which comes from `hibernate-entitymanager` dependency. – Eugene Gotovko Jun 21 '18 at 10:15