7

can i save a file into the local system with the saveAsTextFile syntax ? This is how i'm writing the syntax to save a file: insert_df.rdd.saveAsTextFile("<local path>")

when i'm trying to do this i'm getting error as no permissions, but i have all the permissions to that specific local path, looks like it is treating the file as HDFS file.

Rahul Sharma
  • 5,614
  • 10
  • 57
  • 91
roh
  • 1,033
  • 1
  • 11
  • 19

2 Answers2

17

I think you should try "file:///local path" instead of "/local path".

Simon Schiff
  • 711
  • 6
  • 13
  • i'm running this on cluster, do i have to mention any host name ?? or just the "file:///path" would be enough ?? – roh Oct 24 '16 at 19:57
  • i mean host name before the path – roh Oct 24 '16 at 19:59
  • 4
    I don't use saveAsTextFile, when I want to save the content of an RDD to an text file on my Cluster. Maybe you have to use collect(), but this is not a good Idea on a huge RDD. And if it works, you will get the same number of text Files as the number of Partitions of the RDD. What im am using is in Java the following: `rdd.toLocalIterator().forEachRemaining(x -> bw.write(x.toString())` where bw is a BufferedWriter. – Simon Schiff Oct 24 '16 at 20:08
  • It is similar to [this](http://stackoverflow.com/questions/31239161/save-a-spark-rdd-to-the-local-file-system-using-java) – Simon Schiff Oct 24 '16 at 20:12
-1

The following code works fine:

  val outputFilePath = "file:////home/opsdev/SDG/output/"

  DF.repartition(1)
    .write.mode.option("sep", "|")
    .option("header", "true")
    .option("escape", "\"")
    .option("quoteAll", "true")
    .csv(outputFilePath)

And dont forget to run in local mode.

swapnil shashank
  • 877
  • 8
  • 11