1

I have a current pipeline, where I do several transformations to my dataframe.

It is important to insert checkpoints to assure an accepted execution time.

However from time to time I get this error from any of the checkpoints:

Job aborted due to stage failure: Task creation failed: java.io.FileNotFoundException

No such file or directory: /checkpoints/...../rdd-1002/part-00003

Can you please suggest.

drlol
  • 333
  • 4
  • 18

1 Answers1

0

Is it similar to this issue? A complex logic is overwriting the RDD while a failing process is trying to recover.

Spark not able to find checkpointed data in HDFS after executor fails

Wei Chen
  • 605
  • 6
  • 14
  • yes, but solution given there is not working, also its not an accepted answer – drlol Mar 15 '20 at 07:07
  • You might have to check the data to see what is causing the error first. One possible way to solve this is to have a good `try`, `catch` or `Try` so that the error is handled in the logic to prevent the code block from error since you are just using the checkpoint for time management. – Wei Chen Mar 16 '20 at 08:24