2

I am launching a spark Job from Java application using SparkLauncher. 

SparkAppHandle jobHandle;
try {
    jobHandle = new SparkLauncher()
            .setSparkHome("C:\\spark-2.0.0-bin-hadoop2.7")
            .setAppResource("hdfs://server/inputs/test.jar")
            .setMainClass("com.test.TestJob")
            .setMaster("spark://server:6066")
            .setVerbose(true)
            .setDeployMode("cluster")
            .addAppArgs("abc")
            .startApplication();

} catch (IOException e) {
    throw new RuntimeException(e);
}

while(!jobHandle.getState().isFinal());

I can see my job running on SparkUI and also it is finishing without any errors.

However my java application never terminates since jobHandle.getState() always remains in UNKNOWN state. what am I missing here? My spark API version is 2.0.0. One more detail that might be relevant is that my launcher application is running on windows.

vatsal mevada
  • 5,148
  • 7
  • 39
  • 68
  • Any chance you ever found a solution to this issue? I'm currently running into the same problem. – mongolol Dec 18 '17 at 12:40
  • Unfortunately no. We dropped the plan of using SparkLauncher all together and used spark job-server instead for job submission and management. Also Apache Livy is better alternative of spark job-server as it is incubated by Apache recently. – vatsal mevada Dec 29 '17 at 13:32

1 Answers1

1

You need to block in your main thread and await for a callback from the driver. I've explained a concept in my previous answer.

You can do Thread.sleep in try/catch block, or use Spark Listeners with CountDownLatch.

while(!jobHandle.getState().isFinal()) { 
   //await until job finishes
   Thread.sleep(1000L);
}
Sayat Satybald
  • 6,300
  • 5
  • 35
  • 52