0

I am working on the spark job which updates it status in the table about its exectuion, now If the table contains the successful execution already, I need to stop the spark job gracefully.

I tried doing

System.exit(0)

But the job fails with the

error :Shutdown hook called before final status was reported.

What is the correct procedure to exit the spark job gracefully.

Vishal
  • 1,442
  • 3
  • 29
  • 48
  • 1
    This comment I just left on another thread might be relevant: https://stackoverflow.com/questions/39580449/spark-jobs-running-on-emr-cluster-system-exit0-used-to-gracefully-completion/50872955?noredirect=1#comment107530077_50872955 . Also, my answer to the question explains a possible reason for the error you're seeing. – Ivan Vergiliev Mar 20 '20 at 15:13

1 Answers1

1

You just need to call sc.stop() (on the SparkContext) before exiting your application.

See also similar question on pyspark.

DNA
  • 42,007
  • 12
  • 107
  • 146