3

I'm working with HDFS and Kafka for times, and I note that Kafka is more reliable than HDFS. So working now with Spark-structured-streaming , I'm suprised that checkpointing is only with HDFS. Chekckpointing with Kafka would be faster and reliable. So is it possible to work with spark structured streaming without HDFS ? It seems strange that we have to use HDFS only for streaming data in Kafka. Or is it possible to tell Spark to forget the ChekpPointing and managing it in the program as well ?

Spark 2.4.7

Thank you

DGALLENE
  • 31
  • 1

1 Answers1

1

You are not restricted to use a HDFS path as a checkpoint location.

According to the section Recovering from Failures with Checkpointing in the Spark Structured Streaming Guide the path has to be "an HDFS compatible file system". Therefore, also other file systems will work. However, it is mandatory that all Executors have access to that file system. For example choosing the local file system on the Edge Node in your cluster might be working in local mode, however, in cluster mode this can cause issues.

Also, it is not possible to have Kafka itself handle the offset position with Spark Structured Streaming. I have explained this in more depth in my answer on How to manually set group.id and commit kafka offsets in spark structured streaming?.

Michael Heil
  • 16,250
  • 3
  • 42
  • 77