13

I have a very long Spark job, from which a small number of tasks are currently stalled. Is there any way to kill those stalled tasks from the driver node?

For permission reasons I can log in, but not kill the jobs on the slave nodes, so I'm looking for a way to do this from the driver node alone. Note, that I don't want to kill the entire Spark job - just one or two stalled tasks.

If it helps, I'm using Mesos and have access to web UI, but that does not contain the option to kill the task.

nbubis
  • 2,304
  • 5
  • 31
  • 46

1 Answers1

13

No, not really.

You cannot kill an individual spark task manually, however you can use spark speculation to automatically determine which tasks are taking too long and restart them proactively.

If you want to do that, set spark.speculation to true and [if you dare] modify the spark.speculation.interval, spark.speculation.multiplier, spark.speculation.quantile configuration options.

Related Docs: http://spark.apache.org/docs/latest/configuration.html#viewing-spark-properties

Related SO: How to deal with tasks running too long (comparing to others in job) in yarn-client?

Community
  • 1
  • 1
Garren S
  • 5,552
  • 3
  • 30
  • 45
  • The Spark UI seems to allow you to kill individual tasks and I think they re-schedule and work the next time. This seems to indicate that there is an API for killing tasks. It would probably be a lot of manual work to use it though. I'm suffering through the same issue and speculation is causing other problems currently so I'm debating trying it. – John Humphreys Oct 12 '17 at 16:08
  • 3
    Spark UI lets you kill job stages, but I'm not aware of any functionality to kill individual job tasks. Where is task killing possible in the UI? – Garren S Oct 12 '17 at 17:10
  • You are correct, my apologies. I was using the wrong term and was thinking of stages. – John Humphreys Oct 12 '17 at 19:53