How to store an Array[String] to an output file

Question

I'm having an Array[String] called samparr with some values in it, I want it to get stored as an output file.

var samparr: Array[String] = new Array[String](4)
samparr +:= print1 + "  BEST_MATCH  " + print2

just like,

val output = samparr.saveAsTextFile(outputpath)

but isn't a RDD its an Array[String]

the community get mad, the marked question as duplicate is not for the same purpose as this one — Mehdi Ben Hamida, Mar 15 '18 at 09:12

score 6 · Accepted Answer · edited May 23 '17 at 12:10

6

You can use SparkContext.parallelize to "distribute" your Array onto the Spark cluster (in other words, to turn it into an RDD), and then call saveAsTextFile:

sc.parallelize(samparr).saveAsTextFile(outputpath)

This action will partition the data and send each partition to one of the executors, then each partition will be saved into a separate "file-part".

Alternatively, since the array is very small and doesn't really "justify" using Spark, you can try any non-Spark method of saving data to file, e.g. the one linked by @avihoo-mamka: How to write to a file in Scala?

edited May 23 '17 at 12:10

Community

1
1

answered May 09 '17 at 13:23

Tzach Zohar

37,442
3
79
85

Thank you so much, this is what I expected. Tzach Zohar you Rock.! – Vickyster May 09 '17 at 13:33

How to store an Array[String] to an output file

1 Answers1

Linked