how to kill hadoop jobs

Question

I want to kill all my hadoop jobs automatically when my code encounters an unhandled exception. I am wondering what is the best practice to do it?

Thanks

dr0i · Accepted Answer · 2014-04-28T08:14:39.137

125

Depending on the version, do:

version <2.3.0

Kill a hadoop job:

hadoop job -kill $jobId

You can get a list of all jobId's doing:

hadoop job -list

version >=2.3.0

Kill a hadoop job:

yarn application -kill $ApplicationId

You can get a list of all ApplicationId's doing:

yarn application -list

edited Apr 28 '14 at 08:14

answered Feb 13 '14 at 08:43

dr0i

2,380
2
19
36

2

Also mapred job -kill $jobId – Pradeep Gollakota Apr 12 '15 at 18:09

score 18 · Answer 2 · answered Jul 15 '15 at 22:09

18

Use of folloing command is depreciated

hadoop job -list
hadoop job -kill $jobId

consider using

mapred job -list
mapred job -kill $jobId

answered Jul 15 '15 at 22:09

Prabhu

967
7
14

1

Not working and results in the following error: Exception in thread "main" java.lang.IllegalArgumentException: JobId string : application_1470941878165_0001 is not properly formed – tribbloid Aug 11 '16 at 19:25

score 17 · Answer 3 · answered Oct 14 '16 at 09:16

17

Run list to show all the jobs, then use the jobID/applicationID in the appropriate command.

Kill mapred jobs:

mapred job -list
mapred job -kill <jobId>

Kill yarn jobs:

yarn application -list
yarn application -kill <ApplicationId>

answered Oct 14 '16 at 09:16

Ani Menon

27,209
16
105
126

score 4 · Answer 4 · answered Jul 12 '12 at 20:44

An unhandled exception will (assuming it's repeatable like bad data as opposed to read errors from a particular data node) eventually fail the job anyway.

You can configure the maximum number of times a particular map or reduce task can fail before the entire job fails through the following properties:

mapred.map.max.attempts - The maximum number of attempts per map task. In other words, framework will try to execute a map task these many number of times before giving up on it.
mapred.reduce.max.attempts - Same as above, but for reduce tasks

If you want to fail the job out at the first failure, set this value from its default of 4 to 1.

But this will also cause a job to fail if one node dies and its tasks fail, even if there is no Exception thrown, right? However, I believe that this is indeed the closest solution to what the OP wants. — vefthym, Feb 14 '14 at 13:06

score 2 · Answer 5 · edited Jan 04 '16 at 09:04

2

Simply forcefully kill the process ID, the hadoop job will also be killed automatically . Use this command:

kill -9 <process_id>

eg: process ID no: 4040 namenode

username@hostname:~$ kill -9 4040

edited Jan 04 '16 at 09:04

Cindy Meister

25,071
21
34
43

answered Jan 29 '15 at 02:16

Venu A Positive

2,992
2
28
31

score 1 · Answer 6 · answered Aug 03 '20 at 17:16

Use below command to kill all jobs running on yarn.

For accepted jobs use below command.

for x in $(yarn application -list -appStates ACCEPTED | awk 'NR > 2 { print $1 }'); do yarn application -kill $x; done

For running, jobs use the below command.

for x in $(yarn application -list -appStates RUNNING | awk 'NR > 2 { print $1 }'); do yarn application -kill $x; done

how to kill hadoop jobs

6 Answers6

Linked