3

We use Hazelcast 2.6.2 in a legacy Java clustered application. When the application is stopped the JVM does not terminate any more. It seems that it is caused by Hazelcast threads not being flagged daemon. I did not find a way way through the Hazelcast API to flag them daemon.

Are there recommended solutions to prevent Hazelcast from preventing the JVM to terminate?

Regards

Florian Beaufumé
  • 1,751
  • 2
  • 22
  • 34
  • "when the application is stopped": How? Do you simply end all non-daemon threads or do you actually use `System.exit()`? The latter should always work, no matter if any daemon threads are running or not. – Joachim Sauer Sep 09 '13 at 15:32
  • 4
    Are you calling `hazelcast.shutdownAll();`? – Gray Sep 09 '13 at 15:32
  • @JoachimSauer: all non-daemon threads simply end, otherwise a simple System.exit() would be enough to terminate the JVM like you said – Florian Beaufumé Sep 09 '13 at 16:05
  • @Gray: unfortunatelly I do not know when the application ends, there is no callback for me to execute hazelcast.shutdownAll() or even a System.exit() – Florian Beaufumé Sep 09 '13 at 16:06
  • I've added an answer that shows how to use the shutdown hook. – Gray Sep 09 '13 at 16:08

4 Answers4

2

Looking at the Hazelcast Javadocs, I see that there is a shutdownAll(); method. To quote the javadocs:

Shuts down all running Hazelcast Instances on this JVM, including the default one if it is running. It doesn't shutdown all members of the cluster but just the ones running on this JVM.

If you aren't shutting it down I suspect there are non-daemon threads there that aren't being terminated that will keep the JVM from closing.

Gray
  • 115,027
  • 24
  • 293
  • 354
  • 2
    I think this won't work. The shutdown hook will never get executed since the JVM will not terminate. :'( – Florian Beaufumé Sep 09 '13 at 16:12
  • I think hazelcast.shutdownAll() would allow the JVM to terminate (when the application threads end of course) and so would trigger the shutdown hook. Not the other way around. – Florian Beaufumé Sep 09 '13 at 16:14
  • Duh. You are right @Tom. I'm afraid you are going to have to somehow trigger a call to `shutdownAll()` when the application is quitting. – Gray Sep 09 '13 at 16:17
  • Correct approach. You need to detect somehow that your application is shutting down. If you're using a framework, look for appropriate shutdown hooks and call Hazelcast `shutdownAll()`. – vikingsteve Sep 13 '13 at 19:07
1

In Tomcat, I added a lifecycle listener via server.xml.

<Listener className="com.mycompany.StartupHandler" DEBUG="false"/>

My StartupHandler.class went in a jar in $TOMCAT_HOME/lib/, and it contains this snippet to trigger Hazelcast shutdown when Tomcat shutdown is detected:

public void lifecycleEvent(LifecycleEvent lifecycleEvent) {
    String eventType = lifecycleEvent.getType();
    if (eventType.equals(Lifecycle.BEFORE_STOP_EVENT)) {
        com.hazelcast.core.Hazelcast.shutdownAll();
    }
}

Other web servers should have similar shutdown hooks where you can invoke Hazelcast.shutdownAll() automatically.

See https://github.com/hazelcast/hazelcast/issues/718 for my original Hazelcast thread on the same topic. Thanks for the hint here to call Hazelcast.shutdownAll(). That idea combined with my Tomcat shutdown hook should be sufficient to resolve this problem for me, and hopefully you find it useful too.

Justin Cranford
  • 646
  • 5
  • 7
0

Here is a Tomcat independent way to detect JVM shutdown. It uses a non-daemon thread to poll a daemon thread to detect when the JVM is shutting down (the JVM shuts down daemon threads automatically). Poll the static IS_SHUTDOWN flag in your code, or inline the call to Hazelcast.shutdownAll() here. Be careful of a race condition, because if you try to shutdown Hazelcast before it starts then it throws an exception in your log. The exception does not break anything, but it looks ugly.

/**
 * Non-Daemon thread monitors a sacrificial, low-priority, daemon thread
 * to detect when JVM is shutting down, so shutdown hooks can be invoked.
 * @author justin.cranford
 */
public class ThreadUtil {
    public static boolean IS_SHUTDOWN = false;  // threads can poll this flag, or inline below where flag is set
    private static final Thread DAEMON_THREAD = new Thread() {
        public void run() {
            while (true) {
                try {
                    Thread.sleep(Long.MAX_VALUE);   // sleep forever
                } catch(Exception e) {}
            }
        }
    };
    private static final Thread NONDAEMON_THREAD = new Thread() {
        public void run() {
            while (true) {
                if (!DAEMON_THREAD.isAlive()) { // poll forever, until daemon thread dies
                    ThreadUtil.IS_SHUTDOWN = true;
                    return;
                }
                try {
                    Thread.sleep(1000); // poll every 1000msec = 1sec
                } catch(Exception e) {}
            }
        }
    };
    static {
        DAEMON_THREAD.setName("ShutdownMonitorNonDaemonThread");
        DAEMON_THREAD.setPriority(Thread.MIN_PRIORITY);
        DAEMON_THREAD.setDaemon(true);
        DAEMON_THREAD.start();
        try {
            Thread.sleep(3000); // wait 3000msec = 3sec before monitoring
        } catch(Exception e) {}
        NONDAEMON_THREAD.setName("ShutdownMonitorDaemonThread");
        NONDAEMON_THREAD.setPriority(Thread.MIN_PRIORITY);
        NONDAEMON_THREAD.setDaemon(false);
        NONDAEMON_THREAD.start();
    }
}
Justin Cranford
  • 646
  • 5
  • 7
  • I think this won't work either. The non-daemon thread you use will prevent the JVM from terminating (according to http://geekexplains.blogspot.fr/2008/06/jvm-shutdown-jvm-shutdown-sequence.html ), so the daemon thread will never stop, so we won't detect the "shutdown". – Florian Beaufumé Nov 06 '13 at 10:33
0

As everyone said, hazelcastInstance.shutdownAll() is the solution.

But, I would to get a new feature in Hazelcast - provide a demon client as well. There are many use cases where cache instance needs to be shutdown as soon as application ends.

Sandeep Jindal
  • 14,510
  • 18
  • 83
  • 121