AKKA stops logging stack traces after a while

Question

I've having a really strange issue with Akka and Play 2.3 (Java).

I have a bunch of actors with very standard slf4J logging:

public abstract class ScheduledJob extends UntypedActor {
    protected final LoggingAdapter logger = Logging.getLogger(getContext().system(), this);

My akka.conf looks like this

akka {
         loggers = ["akka.event.slf4j.Slf4jLogger"]
         loglevel = "DEBUG"
                actor {
                  default-dispatcher = {
                     fork-join-executor {
                     parallelism-factor = 4.0
                     parallelism-max = 200
                   }
                 }
               }
          }

My errors look like this for a while:

[2015-06-02 11:25:37,772] - [ERROR] - from akka.actor.OneForOneStrategy at [akka://application/user/distributor-anchor/router]
null
java.lang.NullPointerException: null
        at services.analytics.thirdparty.ServiceImpl.track(ServiceImpl.java:175) ~[tf-rest-play.tf-rest-play-1.0-SNAPSHOT.jar:1.0-SNAPSHOT]
        at actor.services.analytics.DistributorActor.lambda$track$20(DistributorActor.java:146) ~[tf-rest-play.tf-rest-play-1.0-SNAPSHOT.jar:1.0-SNAPSHOT]
        at actor.services.analytics.DistributorActor$$Lambda$25/1308315289.accept(Unknown Source) ~[na:na]
        at java.util.Iterator.forEachRemaining(Iterator.java:116) ~[na:1.8.0_40]
        at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ~[na:1.8.0_40]
        at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580) ~[na:1.8.0_40]
        at actor.services.analytics.DistributorActor.track(DistributorActor.java:145) ~[tf-rest-play.tf-rest-play-1.0-SNAPSHOT.jar:1.0-SNAPSHOT]
        at actor.services.analytics.DistributorActor.onReceive(DistributorActor.java:54) ~[tf-rest-play.tf-rest-play-1.0-SNAPSHOT.jar:1.0-SNAPSHOT]
        at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:167) ~[com.typesafe.akka.akka-actor_2.10-2.3.4.jar:na]
        at akka.actor.Actor$class.aroundReceive(Actor.scala:465) ~[com.typesafe.akka.akka-actor_2.10-2.3.4.jar:na]
        at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:97) ~[com.typesafe.akka.akka-actor_2.10-2.3.4.jar:na]
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) [com.typesafe.akka.akka-actor_2.10-2.3.4.jar:na]
        at akka.actor.ActorCell.invoke(ActorCell.scala:487) [com.typesafe.akka.akka-actor_2.10-2.3.4.jar:na]
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) [com.typesafe.akka.akka-actor_2.10-2.3.4.jar:na]
        at akka.dispatch.Mailbox.run(Mailbox.scala:220) [com.typesafe.akka.akka-actor_2.10-2.3.4.jar:na]
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393) [com.typesafe.akka.akka-actor_2.10-2.3.4.jar:na]
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [org.scala-lang.scala-library-2.10.4.jar:na]

After an indeterminate (8-24 hours) amount of time the stack trace is gone and those same errors are output like this:

[2015-06-02 17:33:43,338] - [ERROR] - from akka.actor.OneForOneStrategy at [akka://application/user/distributor-anchor/router]
 null
 java.lang.NullPointerException: null

I'm 100% sure the above two errors are the same (ie they are caused by the same conditions and thrown in the same place in the code) but can't determine a reason why the stack trace vanishes from the log output after a while. Restart fixes the problem (until it eventually comes back).

score 3 · Accepted Answer · answered Jun 04 '15 at 10:58

3

This is the normal behavior of the JVM, you can switch off the usage of preallocated exceptions by using the -XX:-OmitStackTraceInFastThrow.

answered Jun 04 '15 at 10:58

Roland Kuhn

15,412
2
36
45

Roland you are the man!!!!! Thanks. In case someone else hits this issue here are some helpful links: http://jawspeak.com/2010/05/26/hotspot-caused-exceptions-to-lose-their-stack-traces-in-production-and-the-fix/ http://stackoverflow.com/questions/1076191/nullpointerexception-stack-trace-not-available-without-debug-agent – xtrakBandit Jun 04 '15 at 19:57

AKKA stops logging stack traces after a while

1 Answers1