5

I am trying to understand what will be the impact of adding: spark.yarn.submit.waitAppCompletion=false configuration while submitting spark step. As far as I know, if it is set to true, the client process will stay alive reporting the application's status. Otherwise, the client process will exit after submission.

What will happen when cluster has KeepJobFlowAliveWhenNoSteps = False and submitted step has the spark.yarn.submit.waitAppCompletion=false property? Will the step get killed when it's job not yet completed and rest all other steps completed the execution?

Shaido
  • 27,497
  • 23
  • 70
  • 73
Rony Singh
  • 147
  • 1
  • 10

1 Answers1

1

Based on my experience, with KeepJobFlowAliveWhenNoSteps = False and spark.yarn.submit.waitAppCompletion=false the cluster will be unaware of the status of the current step, so it will think the step is completed and either move to the next step or terminate the cluster.

DanielM
  • 127
  • 1
  • 9