What is the difference between spark checkpoint and local checkpoint? When making local checkpoint I see this in the spark UI:
It shows that local checkpoint is saved on memory.
Asked
Active
Viewed 5,341 times
9
-
1Where do you see "local checkpoint"? – Yuval Itzchakov Nov 14 '19 at 13:43
-
In the image, this storage are relationed with the operations. In the code I use: df.localCheckpoint() – Shadowtrooper Nov 14 '19 at 13:50
2 Answers
8
Local checkpoint stores your data in executors storage (as shown in your screenshot). It is useful for truncating the lineage graph of an RDD, however, in case of node failure you will lose the data and you need to recompute it (depending on your application you may have to pay a high price).
'Standard' checkpoint stores your data in a reliable file system (like hdfs). It is more expensive to perform but you will not need to recompute the data even in case of failures. Of course, it truncates the lineage graph.
Truncating a long lineage graph avoid getting stack overflow exceptions and is particularly useful in iterative algorithms
LizardKing
- 601
- 6
- 13
-
5For me local checkpointing write directly to executors *disks*, not memory. The [doc](https://spark.apache.org/docs/2.4.1/api/scala/index.html#org.apache.spark.sql.Dataset) says: "Local checkpoints are written to executor storage" – bonnal-enzo Nov 14 '19 at 14:41
4
- local checkpointing writes data in executors storage
- regular checkpointing writes data in HDFS
local checkpointing is faster than classic checkpointing but regular checkpointing is safer in that it leverages HDFS reliability (e.g. data blocks replication).
bonnal-enzo
- 1,165
- 9
- 19