how to set checkpiont dir PySpark Data Science Experience

Question

Could you help me with instructions on how to set the checkpoint dir for a PySpark session on IBM's Data Science Experience?.

The need came because i have to run connectedComponents() from GraphFrames and it raises the following error

Py4JJavaError: An error occurred while calling o221.run.
: java.io.IOException: Checkpoint directory is not set. Please set it first using sc.setCheckpointDir().

Have you tried `sc.setCheckpointDir()`? The error message is pretty clear about that, isn't it? — Roland Weber, Mar 08 '18 at 06:35
yes, however I did not know which specific directory should I put. — ElBrocas, Mar 09 '18 at 15:31

score 10 · Answer 1 · answered Mar 09 '18 at 15:38

The main issue is to get the directory that the notebook has as working directory to set the checkpoit dir with sc.setCheckpointDir(). this can be done easily with

!pwd

Then, a directory for checkpoints should be created on that route

!mkdir <pwd_output>/checkpoints

Finally set the checkpoint

spark.sparkContext.setCheckpointDir('<pwd_output>/checkpoints')

how to set checkpiont dir PySpark Data Science Experience

1 Answers1