Automatically delete checkpoint files in PySpark

Question

I'm creating spark context and session in my PySpark code like this,

conf = SparkConf().set("spark.cleaner.referenceTracking.cleanCheckpoints", "true")
sc = SparkContext.getOrCreate(conf=conf)

spark = SparkSession(sc)
spark.sparkContext.setCheckpointDir("../../checkpoints")

In my code that follows I'm using checkpoint() over some dataframes. It works as expected.

But I want to remove the checkpoints after the code is run to completion.

Is there a spark configuration that I can use? cleanCheckpoints is not doing that.

How can I delete those checkpoint files when the code is completed? What is the best approach?

Related: https://stackoverflow.com/q/60009856/877069 and https://stackoverflow.com/q/52630858/877069 — Nick Chammas, Mar 10 '21 at 20:32

score 0 · Answer 1 · answered Apr 27 '20 at 08:55

0

Write any cleanup logic inside onApplicationStart or onApplicationEnd method of SparkListener & take look at SparkListener abstract class for other useful methods.

Note: Below scala code shows how to register SparkLister & access methods.


spark.sparkContext.addSparkListener(new SparkListener() {

  override def onApplicationStart(applicationStart: SparkListenerApplicationStart) {
    println("Spark ApplicationStart: " + applicationStart.appName);
  }
  override def onApplicationEnd(applicationEnd: SparkListenerApplicationEnd) {
    println("Spark ApplicationEnd: " + applicationEnd.time);
  }
});

answered Apr 27 '20 at 08:55

Srinivas

8,957
2
12
26

How can I do this in PySpark? – Sreeram TP Apr 27 '20 at 09:02
This post will help you to achieve same - https://stackoverflow.com/questions/44082957/how-to-add-a-sparklistener-from-pyspark-in-python – Srinivas Apr 27 '20 at 09:07
I think it's bit complex to do same in python..:) – Srinivas Apr 27 '20 at 09:17

Automatically delete checkpoint files in PySpark

1 Answers1