3

I have a web application which will submit spark jobs on Cloudera spark cluster using spark launcher library.

It is successfully submitting the spark job to cluster. However it is not calling back the listener class methods and also the getState() on returned SparkAppHandle never changes from "UNKNOWN" even after job finishes execution on cluster.

I am using yarn-cluster mode. Here is my code. Is anything else needs to be done?

SparkLauncher launcher = new SparkLauncher()
                           .setSparkHome("sparkhome")
                           .setMaster("yarn-cluster")
                           .setAppResource("spark job jar file")
                           .setMainClass("spark job driver class")
                           .setAppName("appname")
                           .addAppArgs(argsArray)
                           .setVerbose(true)
                           .addSparkArg("--verbose");

SparkAppHandle handle = launcher.startApplication(new LauncherListener());

int c = 0;
while(!handle.getState().isFinal()) {
  LOG.info(">>>>>>>> state is=  "+handle.getState() );
  LOG.info(">>>>>>>> state is not final yet. counter=  "+c++ );
  LOG.info(">>>>>>>> sleeping for a second");
  try {
    Thread.sleep(1000L);
  } catch (InterruptedException e) {
  }
  if(c == 200)
    break;
}

Here are the things I have already tried:

  1. Added listener instance to SparkAppHandle once application is launched.
  2. Made the current class implement SparkAppHandle.Listener and passed it (this) in both ways (while launching, and by setting it on SparkAppHandle)
  3. Tried to use launcher.launch() method so that at least I can block on the resulting Process object by calling process.waitFor() method till spark job finishes running on cluster. However in this case, for long running spark jobs, corresponding process on this node never returns (though it works fine for spark jobs which are finishing in 1 or 2 min)
Reddy
  • 8,737
  • 11
  • 55
  • 73
  • Where you able to solve this? – msemelman Sep 23 '16 at 14:51
  • 2
    The reply I got on spark user mailing list is that underlying spark needs to be at least 1.6.0 version (I am using 1.5.1). Once I successfully tests it, I will post it as an answer. – Reddy Sep 23 '16 at 18:13

1 Answers1

0

I have got answer for this question from spark user mailing list. For this functionality to work, not only spark launcher needs to be 1.6.0 but also underlying spark should be 1.6.0 at least.

I have been using spark 1.5.1 and 1.6.0 version of launcher library. Now I have updated spark cluster to 1.6.0 version and now I am getting the callbacks to listener methods.

Reddy
  • 8,737
  • 11
  • 55
  • 73
  • What had to be done to get this working? I'm also facing the same issue. Application exits immediately with getting appID and state. Works if I call Thread.sleep() explicitly. Thank you! – – Tariq Nov 10 '16 at 20:58
  • The issue I faced was not getting state or appID at all. If you want to wait for spark job to finish, I think, you can put a loop which will poll status of the handle till it reaches a final state (`handle.getState().isFinal()`). – Reddy Nov 11 '16 at 05:43