1

Using Spark 1.5 Streaming with an Actor receiver. 

val conf = new SparkConf()
    .setMaster("local[4]")
    .setAppName("ModelTest")

val ssc = new StreamingContext(conf, Seconds(2))

val models = ssc.actorStream[Model](Props(...), "ModelReceiver")

models.foreachRDD { rdd => ... }

ssc.start()
ssc.awaitTermination()
// NEVER GETS HERE!

When the generated Actor is shutdown the code will not progress beyond ssc.awaitTermination()

If I kill SBT with Ctrl+C a println after the ssc.awaitTermination() line will complete.

How should Spark be terminated?

BAR
  • 15,909
  • 27
  • 97
  • 185

1 Answers1

2

You are correct that Spark Streaming will await termination, as the function name hints. To kill a Streaming application you send a SIGTERM to that process, for example by using the kill command.

As you can also see in the Spark Standalone documentation you can also kill the process using Spark Submit:

./bin/spark-class org.apache.spark.deploy.Client kill <master url> <driver ID>

You can define some code that you want to run when the process is shutting down, by calling sys.ShutdownHookThread. 

sys.ShutdownHookThread {
  log.info("Stopping Spark Streaming...")
  ssc.stop(stopSparkContext = true, stopGracefully = true)
  log.info("Shutting down the application...")
}
Patrick McGloin
  • 2,204
  • 1
  • 14
  • 26
  • I apologize if i'm missing something obvious, but I don't see how this answers my question - how do I exit awaitTermination()? – BAR Sep 28 '15 at 08:31
  • You terminate the process, either as you are doing with CTRL-C, using kill and SIGTERM or using "spark-class ... kill". And then the process will finish. – Patrick McGloin Sep 28 '15 at 08:34