5

My Java application runs on mapper and creates child processes using Qubole API. Application stores child qubole queryIDs. I need to intercept kill signal and shutdown child processes before exit. hadoop job -kill jobId and yarn application -kill applicationId commands are killing job in a SIGKILL manner, I do not know how to intercept shutdown. Is it possible to intercept job kill somehow or configure hadoop to give application chance to shutdown gracefully?

Application successfully intercepts shutdown using ShutdownHook when running locally, not in mapper container and able to kill it's child processes.

Please suggest how to intercept shutdown when running in mapper, or maybe I'm doing something wrong?

leftjoin
  • 36,950
  • 8
  • 57
  • 116
  • Regarding ShutdownHooks not being executed, it is expected if the application is killed with a SIGKILL, (see: https://stackoverflow.com/questions/2541597/how-to-gracefully-handle-the-sigkill-signal-in-java). I don't believe you can "handle" a SIGKILL, as mentioned in the link, you can monitor the state of the job. As far as I know no tools allow for a "graceful shutdown" of jobs so far. – Adonis Jul 13 '17 at 13:27

1 Answers1

4

SIGKILL is unstoppable and no process can catch it. Neither your Java application, neither the JVM itself... It is, in fact, not event sent to the process. Consider it more as a direct order to the kernel to destroy all the process resources without delay.

From man 7 signal:

the signals SIGKILL and SIGSTOP cannot be caught, blocked, or ignored.

This is a native core kernel feature, you cannot bypass it.

Also note that according to Prabhu (2015-07-15) on how to kill hadoop jobs:

Use of following command is depreciated

hadoop job -list
hadoop job -kill $jobId

consider using 

mapred job -list
mapred job -kill $jobId

This is verified on the Apache Hadoop - Deprecated API Documentation

Unfortunately, according to the current mapred Command Documentation it does not appear that you can control the type of signal sent to terminate a job.

Fabien
  • 4,862
  • 2
  • 19
  • 33
  • So I cannot control the type of signal sent to terminate a job and cannot shutdown gracefully. – leftjoin Jul 19 '17 at 18:09
  • Well, I do not say that you cannot, but obviously the ways to do so are not standards and will involve (a lot) more code/scripts to maintain... Most probably ends up in a DIY solution. – Fabien Jul 19 '17 at 18:42