Spark: Saving RDD in an already existing path in HDFS

Question

I am able to save the RDD output to HDFS with saveAsTextFile method. This method throws an exception if the file path already exists.

I have a use case where I need to save the RDDS in an already existing file path in HDFS. Is there a way to do just append the new RDD data to the data that is already existing in the same path?

score 7 · Answer 1 · answered Jul 29 '16 at 16:41

7

One possible solution, available since Spark 1.6, is to use DataFrames with text format and append mode:

val outputPath: String = ???

rdd.map(_.toString).toDF.write.mode("append").text(outputPath)

answered Jul 29 '16 at 16:41

zero323

322,348
103
959
935

I tried it. But, I got this exception java.lang.NoSuchMethodError: org.apache.spark.sql.DataFrameWriter.text(Ljava/lang/String;)V – yAsH Jul 29 '16 at 18:59
Which version of Spark do you use? – zero323 Jul 29 '16 at 19:01
Oops. I am using spark version 1.5. – yAsH Jul 29 '16 at 19:03
In 1.5 you could do similar thing with [spark-csv](https://github.com/databricks/spark-csv) and adjusting `header` and `quoteMode` – zero323 Jul 29 '16 at 19:05

Spark: Saving RDD in an already existing path in HDFS

1 Answers1

Linked