1

I'm using Spark version 1.3. I have a job that's taking forever to finish. To fix it, I made some optimizations to the code, and started the job again. Unfortunately, I launched the optimized code before stopping the earlier version, and now I cannot stop the earlier job.

Here are the things I've tried to kill this app:

  1. Through the Web UI result: The spark UI has no "kill" option for apps (I'm assuming they have not enabled the "spark.ui.killEnabled", I'm not the owner of this cluster).

  2. Through the command line: spark-class org.apache.spark.deploy.Client kill mymasterURL app-XXX

result: I get this message:

Driver app-XXX has already finished or does not exist

But I see in the web UI that it is still running, and the resources are still occupied.

  1. Through the command line via spark-submit: spark-submit --master mymasterURL --deploy-mode cluster --kill app-XXX

result: I get this error:

Error: Killing submissions is only supported in standalone mode!

  1. I tried to retrieve the spark context to stop it (via SparkContext.stop(), or cancelAllJobs() ) but have been unsuccessful as ".getOrCreate" is not available in 1.3. I have not been able to retrieve the spark context of the initial app.

I'd appreciate any ideas!

Edit: I've also tried killing the app through yarn by executing: yarn application -kill app-XXX

result: I got this error:

Exception in thread "main" java.lang.IllegalArgumentException: Invalid ApplicationId prefix: app-XX. The valid ApplicationId should start with prefix application

user2662165
  • 189
  • 1
  • 1
  • 10
  • Even a command that would kill all running jobs on the Spark cluster would work for me but I can't find such a command. – user2662165 Dec 31 '16 at 19:14
  • How did you execute spark (command line)? is it spark standalone? yarn-client, etc? – Yaron Dec 31 '16 at 19:58
  • I have not executed spark and I'm only a user, someone else is in charge of maintaining this cluster. However, It is definitely not in standalone mode (see my comment above about the error I got from Killing submissions). I'm trying to find out if yarn-client is used. – user2662165 Dec 31 '16 at 20:30
  • 3
    With the message that you received for #3 (`Error: Killing submissions is only supported in standalone mode!`), would guess that you have yarn-client. If that is the case, then you should be able to run the command `yarn application -kill app-XXX` (more info at: http://stackoverflow.com/a/30253320/1100699) – Denny Lee Jan 01 '17 at 05:11
  • @Danny Lee You are correct about yarn-client, but this command also did not work. Here's what I did that worked: I tracked down the driver process and the machine that was running it, then I logged into that particular machine, got a list of processes, and found the pyspark-shell related to my program (based on the start time of my process) and killed it. Of course, I don't recommend such a coarse and hacky way of killing apps but I was really cornered into this situation. – user2662165 Jan 02 '17 at 02:28
  • yarn is failing because it is not the application id, you need list your applications first using 'yarn application list' and identify your spark app (application id will be similar to application-id-xxxxxxxxxxxxx) and that is the app-id you need to use in your command. – MrElephant Mar 17 '20 at 20:05

0 Answers0