WebApplicationContext not being shut down on Servlet Context reload

Question

When I shut down Tomcat, I observe a correct shutdown and cleanup of the Spring WebApplicationContext. However, when I redeploy my Spring-based WAR (by copying the new WAR to webapps), normal shutdown does not occur. This is a problem for me due to all the ensuing resource leaks:

org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [] appears to have started a thread named [hz.hazelcast-swipe-instance.scheduled] but has failed to stop it. This is very likely to create a memory leak.
org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [] appears to have started a thread named [hz.hazelcast-swipe-instance.operation.thread-0] but has failed to stop it. This is very likely to create a memory leak.

... and many more. I am using XML-less configuration, this is my WebApplicationInitializer:

public class WebApplicationInitializer extends AbstractAnnotationConfigDispatcherServletInitializer
{
  @Override protected Class<?>[] getRootConfigClasses() {
    return new Class[] { WebSecurityConfig.class, WebMvcConfig.class };
  }
  @Override protected Class<?>[] getServletConfigClasses() { return null; }

  @Override protected String[] getServletMappings() { return new String[] { "/" }; }

  @Override public void onStartup(ServletContext ctx) throws ServletException {
    ctx.setInitParameter("spring.profiles.active", "production");
    super.onStartup(ctx);
  }
}

There is no configuration specific to controlling the behavior upon servlet context reload, and I assume this should have worked out of the box.

Is there a way to make the WebApplicationContext close properly before continuing the servlet context reloading procedure?

I am on Spring 4.0.5, Tomcat 7.0.54, Hazelcast 3.2.1, Hibernate 4.3.4.Final.

Update

I have added a Spring application listener for the ContextClosedEvent and printed the stack trace of its invocation:

    at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:333) [spring-context-4.0.6.RELEASE.jar:4.0.6.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:335) [spring-context-4.0.6.RELEASE.jar:4.0.6.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:880) [spring-context-4.0.6.RELEASE.jar:4.0.6.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:841) [spring-context-4.0.6.RELEASE.jar:4.0.6.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.destroy(FrameworkServlet.java:819) [spring-webmvc-4.0.6.RELEASE.jar:4.0.6.RELEASE]
    at org.apache.catalina.core.StandardWrapper.unload(StandardWrapper.java:1486) [catalina.jar:7.0.54]
    at org.apache.catalina.core.StandardWrapper.stopInternal(StandardWrapper.java:1847) [catalina.jar:7.0.54]
    at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:232) [catalina.jar:7.0.54]
    at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5647) [catalina.jar:7.0.54]
    at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:232) [catalina.jar:7.0.54]
    at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1575) [catalina.jar:7.0.54]
    at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1564) [catalina.jar:7.0.54]

This indicates that the Spring shutdown occurs in its Servlet#destroy method. This is the relevant snippet from AbstractApplicationContext#close():

        if (logger.isInfoEnabled()) {
            logger.info("Closing " + this);
        }

        LiveBeansView.unregisterApplicationContext(this);

        try {
            // Publish shutdown event.
            publishEvent(new ContextClosedEvent(this));
        }
        catch (Throwable ex) {
            logger.warn("Exception thrown from ApplicationListener handling ContextClosedEvent", ex);
        }
        // Stop all Lifecycle beans, to avoid delays during individual destruction.
        try {
            getLifecycleProcessor().onClose();
        }
        catch (Throwable ex) {
            logger.warn("Exception thrown from LifecycleProcessor on context close", ex);
        }

        // Destroy all cached singletons in the context's BeanFactory.
        destroyBeans();

        // Close the state of this context itself.
        closeBeanFactory();

        // Let subclasses do some final clean-up if they wish...
        onClose();

        synchronized (this.activeMonitor) {
            this.active = false;
        }

I see the log entry from the start of this snippet, and I get my ContextClosedEvent. I also see an entry DefaultLifecycleProcessor - Stopping beans in phase 2147483647, which probably comes from the getLifecycleProcessor.onClose() line. It seems that some error occurs downstream from that. Some exception may be swallowed.

Update 2

As requested, this is how I configure Hazelcast:

@Bean(destroyMethod="shutdown") public HazelcastInstance hazelcast() {
  final Config c = hzConfig();
  final JoinConfig join = c.getNetworkConfig().getJoin();
  join.getMulticastConfig().setEnabled(false);
  join.getTcpIpConfig().setEnabled(true);
  return getOrCreateHazelcastInstance(c);
}

hzConfig() is a method where instance name, group name and password, map names, and map indices are configured, so I don't think it is of interest here.

And this is my Hibernate SessionFactory config:

@Bean
public LocalSessionFactoryBean sessionFactory() {
  final LocalSessionFactoryBean b = new LocalSessionFactoryBean();
  b.setDataSource(dataSource);
  b.setHibernateProperties(props(
      "hibernate.connection.release_mode", "on_close",
      "hibernate.id.new_generator_mappings", "true",
      "hibernate.hbm2ddl.auto", "update",
      "hibernate.order_inserts", "true",
      "hibernate.order_updates", "true",
      "hibernate.max_fetch_depth", "0",
      "hibernate.jdbc.fetch_size", "200",
      "hibernate.jdbc.batch_size", "50",
      "hibernate.jdbc.batch_versioned_data", "true",
      "hibernate.jdbc.use_streams_for_binary", "true",
      "hibernate.use_sql_comments", "true"
  ));
  return b;
}

Answer 1

At some point, you mentioned that there was a NoClassDefFoundError for Logback. You got this fixed by removing this dependency, but then the problem moved to a another class - one of Spring's own classes.

This can mean that either one of the libraries you have does something buggy with class loaders or maybe Tomcat needs instructions not to keep locks on some resources. See here more about Tomcat resources being locked and the <Context> setting to try: in your Tomcat's conf/context.xml place a antiResourceLocking="true" to the element.

Answer 2

Have you tried upping unloadDelay (defaults to 2000ms) for Tomcat contexts? See http://tomcat.apache.org/tomcat-7.0-doc/config/context.html

UPDATE: I see that you are having issues with logback as well, it might be worth the shot to try and register this listener as well:

class LogbackShutdownListener implements ServletContextListener {

    @Override
    public void contextDestroyed(ServletContextEvent event) {
        LoggerContext loggerContext = (LoggerContext)LoggerFactory.getILoggerFactory();
        System.out.println("Shutting down Logback context '" + loggerContext.getName() + "' for " + contextRootFor(event));
        loggerContext.stop();
    }

    @Override
    public void contextInitialized(ServletContextEvent event) {
        System.out.println("Logback context shutdown listener registered for " + contextRootFor(event));
    }

    private String contextRootFor(ServletContextEvent event) {
        return event.getServletContext().getContextPath();
    }

}

Be sure to declare this listener before the spring context loader listener so that it is invoked after the context listener upon shutdown.

UPDATE 2: Also it might be worth the try to register another bean to handle closing of the Hazelcast stuff manually (be sure to also remove destroyMethod from the hazelcast bean):

@Component
class HazelcastDestructor {

    @Autowired
    private HazelcastInstance instance;

    @PreDestroy
    public void shutdown() {
        try {
            instance.shutdown();
        } catch (Exception e) {
            System.out.println("Hazelcast failed to shutdown(): " + e);
            throw e;
        }
    }  
}

UPDATE 3: Just out of curiosity, have you tried parallel deployment: http://www.javacodegeeks.com/2011/06/zero-downtime-deployment-and-rollback.html. It might behave differently than reloading the very same context. At the very least you should be able to undeploy the old version lazily and see if that makes a difference.

Answer 3

There is a similar issue on the dangling threads while container restarting here.

Of all the answers, one particular answer of interest was by Howard - which shows the way these threads are cleared.

There is some good discussion and reasoning as to how this can terminate the threads here.

Now implement ServletContextListener and take care of these threads in the contextDestroyed() method as:

    public class YourListener implements ServletContextListener{
        ....
        @Override
        public void contextDestroyed(ServletContextEvent event) {
            //Call the immolate method here
        }
    }

Register this listener in WebApplicationInitilizer as:

     ctx.addListener(new YourListener());

So when server is restarted - contextDestroyed method is called and this takes care of all these threads.

Answer 4

From Web App development point of view, ServletContainer can only notify the before started and before end process of app. It is using ServletContextListener.

Config ServletContextListener in web.xml

<listener>
    <listener-class>com.var.YourListener</listener-class>
</listener>

YourListener.java

public class YourListener implements ServletContextListener {

    public void contextInitialized(ServletContextEvent sce) {
        //initialization process
    }
    public void contextDestroyed(ServletContextEvent sce) {
        //destory process
    }
}   

Update -XML Less

Programmatically

@Override 
public void onStartup(ServletContext ctx) throws ServletException {

    ctx.addListener(new YourContextListener());

    ctx.setInitParameter("spring.profiles.active", "production");
    super.onStartup(ctx);
}   

Annotation

@WebListener / @WebServletContextListener 
public class YourContextListener implements ServletContextListener {

    public void contextInitialized(ServletContextEvent servletContextEvent) {
    }

    public void contextDestroyed(ServletContextEvent servletContextEvent) {
    }
}

Update- ShoutDown Hook In Spring

I never use it before my app development, we can register shoutdownhook event into AbstractApplicationContext of Spring. I am not sure it will be ok or not for you.

AbstractApplicationContext context = ...
context.registerShutdownHook();

Reference 1 Reference 2