0

I'm a beginner in spark streaming. I'm producing and consuming messages using spark streaming. But I cannot stop the consumer . Actually I need to stop the streaming when the producer sends no message. Could somebody help me on that with java code. If possible give me a small example codes to stop the streaming on java.

I'm using spark version 2.2.0

conf.set("spark.streaming.stopGracefullyOnShutdown","true");
// Execute the Spark workflow defined above
streamingContext.start();
try {
    streamingContext.awaitTermination();
} catch (Exception e) {
    // e.printStackTrace();
}
Suresh Kumar
  • 172
  • 3
  • 12
  • 1
    Have a look at [this](https://stackoverflow.com/questions/39990681/how-to-stop-running-spark-streaming-application-gracefully/39993190#39993190) answer. – Glennie Helles Sindholt Aug 17 '17 at 07:22
  • I have updated with **conf.set("spark.streaming.stopGracefullyOnShutdown","true");** But it is not stopping the streaming . Could you help me on this? @Glennie – Suresh Kumar Aug 17 '17 at 08:37
  • Are you sending the stop signal as described in the post?`stopGracefullyOnShutdown` only ensures that the streaming job is stopped gracefully when the JVM receives a stop-signal - you have to send the signal (or you have to write a program that sends the signal). – Glennie Helles Sindholt Aug 17 '17 at 08:44
  • I'm a beginner. I don't know how to send the stop signal. Could you help me on that? Or could you show me any simple code to do that? @ Glennie – Suresh Kumar Aug 17 '17 at 08:50
  • The post tells you the actual command you can use to stop the streaming. I'm not sure what else you need... – Glennie Helles Sindholt Aug 17 '17 at 08:58
  • Actually in that blog they are passing the signal from the shell script. But I'm using a java program and I need to hard code the signal to stop the stream on particular condition. Actually what I need is when all the message from the producer is consumed (There is no message from the producer) then my consumer gets zero lines. At that time the stream should stop looking for the producer port. How to do that? – Suresh Kumar Aug 17 '17 at 09:04
  • It seems to me that you are mixing things up. A streaming program is - well a program that handles a continuos stream. It runs all the time, processing the data as it comes in, and isn't stopped whenever there is no data in the stream. Stopping a streaming app should only be necessary when you need to upgrade software or something along those lines. If you want to just have a program read from a queue every once in a while, don't write a streaming app, write a traditional program that runs on a schedule to process the messages on the queue and then shuts down. – Glennie Helles Sindholt Aug 17 '17 at 09:34
  • Yes. Finally got the clear view about the purpose of streaming VS normal programming. Thank you for your help. @Glennie – Suresh Kumar Aug 17 '17 at 10:04

1 Answers1

0

The stream does not know whether the data is end. Perhaps you can kill the jobif you are sure the data is end.

$spark-submit --master $masterurl --kill $driverid

You can find the driverid by the jobWEBUI.

Indrajit Swain
  • 1,505
  • 1
  • 15
  • 22
Robin
  • 695
  • 1
  • 7
  • 10