3

I am trying to understand the spark UI to monitor the timings but having difficulty in understanding job duration and task duration relationship.

For Below jobs it says total run time 13 Min , but when i open the stage (which have 1 stage and 1 task) within job, its saying just 3 Seconds. How Can i get the details where other time is being spent.

Thanks in Advance!!!

enter image description here

enter image description here

gkarya42
  • 429
  • 6
  • 22
  • Below answers might help you in understanding this : https://stackoverflow.com/questions/40466265/spark-why-the-spark-job-duration-is-not-equal-to-the-sum-of-each-stage-duratio?rq=1 https://stackoverflow.com/questions/42263270/what-is-the-concept-of-application-job-stage-and-task-in-spark?rq=1 – Anil Kumar Feb 17 '22 at 08:23

0 Answers0