Spark. Recover checkpointed RDD after driver shutdown

Asked Oct 28 '19 at 09:38

Active Oct 28 '19 at 09:38

Viewed 46 times

I checkpointed a rdd which takes very long to compute. Then I executed many jobs on such a rdd. Eventually, one of this job failed and the driver shutdown during night. Now I need to recover the checpointed data but I can't. There are many questions like this in SO but none of them answer the question. f.e.:

How to read checkpointed RDD <= The only answer replicates the documentation. Which Is useless

How to recover from checkpoint when using python spark direct approach? <= Is about streaming context.

My enviroment is azure databricks notebooks spark 2.4.3 and python 3

asked Oct 28 '19 at 09:38

lsmor

4,698
17
38

Certainly on topic. – thebluephantom Oct 28 '19 at 20:56
That might be fatal? What about this Scala question of mine? https://stackoverflow.com/questions/57840083/spark-checkpointing-non-streaming-checkpoint-files-can-be-used-in-subsequent-j If u like, upvote it – thebluephantom Oct 28 '19 at 21:00

Spark. Recover checkpointed RDD after driver shutdown

0 Answers0