after deploying my Spark Streaming Job on a Standalone Spark cluster, i got some problems with checkpointing. The console log yields a hint:
WARN ReliableCheckpointRDD: Error writing partitioner org.apache.spark.HashPartitioner@2 to hdfs://10.254.25.21:8020/path/1914a5db-96be-4634-b2ce-ee867119fd95/rdd-18129
I am using the default HashPartitioner dividing the data in two partitions. I set my HDFS checkpointing directory to my Spark master and HDFS port as follows:
ssc.checkpoint("hdfs://10.254.25.21:8020/path")
In my Job I never manually call .checkpoint(duration) on any DStream myself. But I got many stateful Streams resulting from PairDStreams mapWithState() invokations. The code of catching the exception can be found in ReliableCheckpointRDD line 209ff. Unfortunately, I could not find any references to this error on the web.
In my job, the exception is thrown for every stateful DStream whenever checkpointing is triggered.
Any help is appreciated!
Edit #1
This does not affect the correctness of the results. Yet I wonder if the performance is getting worse as I am doing some performance analysis.
