Akka not shutting down cleanly in Servlet

Question

I'm developing a Servlet-based web application in Scala, and using Akka. Everything works fine while it's up and running, I see no errors when viewing code, my child actors all bring themselves up and shut themselves down correctly. However; when I try to shut down my server, or re-deploy I get a lot of errors in the console:

Shutting down...
Shut down successfully.
27-Mar-2015 11:21:22.233 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [ROOT] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@68d6d1aa]) and a value of type [scala.concurrent.forkjoin.ForkJoinPool.Submitter] (value [scala.concurrent.forkjoin.ForkJoinPool$Submitter@1045f98]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
27-Mar-2015 11:21:22.233 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [ROOT] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [scala.util.DynamicVariable$$anon$1@5f818593]) and a value of type [org.apache.tomcat.util.log.SystemLogHandler] (value [org.apache.tomcat.util.log.SystemLogHandler@5b616a23]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
27-Mar-2015 11:21:22.241 INFO [main] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["http-nio-8080"]
27-Mar-2015 11:21:22.243 INFO [main] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["ajp-nio-8009"]
27-Mar-2015 11:21:22.244 INFO [main] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["http-nio-8080"]
27-Mar-2015 11:21:22.244 INFO [main] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["ajp-nio-8009"]
Disconnected from server

If I try use a regular, synchronous servlet I don't get the messages in the log upon shutdown.

I have a context listener on the servlet which is the following:

class ContextListener extends ServletContextListener {
    private var system: ActorSystem = _

    override def contextInitialized(sce: ServletContextEvent): Unit = {
        val context = sce.getServletContext

        system = ActorSystem.create("StridentStandard")

        context.setAttribute("actor.system", system)
    }

    override def contextDestroyed(sce: ServletContextEvent): Unit = {
        println("Shutting down...")
        system.shutdown()
        system.awaitTermination()
        println("Shut down successfully.")
    }
}

So, from what it looks like - the ActorSystem should be shutting down correctly - but it seems some threads are hanging on.

I'm fairly new to Scala, and Akka... and concurrency, and as a result am not really sure where to go from here.

use jstack to get a list of the threads that are hanging around, from there we can figure out what to do about each of them — Chris K, Mar 27 '15 at 13:54

score 1 · Answer 1 · edited May 23 '17 at 12:14

1

Try to set in your application.conf

akka {
  daemonic=on
}

As in that questions - I had the same problem and it helped

edited May 23 '17 at 12:14

Community

1
1

answered Mar 27 '15 at 11:54

M4ks

11,744
7
27
48

I've added the configuration value, but it hasn't had any effect. How can I be sure it's loaded? – Seer Mar 27 '15 at 12:04
I'm not sure, if you rebuilt/repacked application it should "just work" – M4ks Mar 27 '15 at 12:12
Hmmmm, I've checked that the config value is loaded, but it's not making any difference. – Seer Mar 27 '15 at 12:34

Chris K · Answer 2 · 2015-03-27T14:06:25.770

The key to your problem is in this log line

27-Mar-2015 11:21:22.233 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [ROOT] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [scala.util.DynamicVariable$$anon$1@5f818593]) and a value of type [org.apache.tomcat.util.log.SystemLogHandler] (value [org.apache.tomcat.util.log.SystemLogHandler@5b616a23]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

Upon webapp unloading, Tomcat was getting so many reports of resource leaks that were due to the webapp and not the tomcat container that they implemented this check. The error is not reporting that akka threads have been left running, but that a value stored in a ThreadLocal has not been cleared up. This will prevent the JVM from GCing the webapp when the webapp is unloaded.

The culprit is DynamicVariable, a scala class that wraps ThreadLocal. To remove this error, you will need to track down who is using this code and make it clean itself up. If it is akka code responsible, then try upgrading to a later version and if the problem still continues consider submitting a patch to the akka team.

Tomcat documents their memory leak protection protocols here.

Akka not shutting down cleanly in Servlet

2 Answers2