Spark Map transformation

Question

Below code will extract data from file and splitting data with comma delimiter.

val a=sc.textFile("/sparkinput")
val b=a.map(x=>x.split(","))
b.collect

Output:

Array(Array(1, Brandon Buckner, avil, female, 525), Array(2, Veda Hopkins, avil, male, 633), Array(3, Zia Underwood, paracetamol, male, 980), Array(4, Austin Mayer, paracetamol, female, 338), Array(5, Mara Higgins, avil, female, 153), Array(6, Sybill Crosby, avil, male, 193), Array(7, Tyler Rosales, paracetamol, male, 778), Array(8, Ivan Hale, avil, female, 454), Array(9, Alika Gilmore, paracetamol, female, 833), Array(10, Len Burgess, metacin, male, 325))


b.saveasTextFile("/sparkoutput")

When I save output in textfile not sure why i am getting data as below.
Output:

[Ljava.lang.String;@45968fc2

[Ljava.lang.String;@6b3012a2

[Ljava.lang.String;@211780a8

[Ljava.lang.String;@dffa691

[Ljava.lang.String;@b1080c5

[Ljava.lang.String;@68760b2f

[Ljava.lang.String;@1dd8703d

I am new to spark,please help me on this.

score 1 · Answer 1 · edited May 23 '17 at 12:06

1

I think that when using "saveasTextFile" on array of array - this might result in writing Java object instead of string.

I think that using flatMap instead of map might help you get strings in your output-file.

see: https://stackoverflow.com/a/22510434/5088142

edited May 23 '17 at 12:06

Community

1
1

answered Oct 27 '16 at 07:20

Yaron

10,166
9
45
65

I got the expected result when i use flatmap but i am so curious to know why i didnt get expected out for map.Now i got the answer ..thanks!! – vivman Oct 27 '16 at 07:31
@Yaron if I may ask about your suggestion to replace `collect` with `collect()` , why ? – eliasah Oct 28 '16 at 07:21
@eliasah - I've removed my comment about collect vs collect() – Yaron Oct 30 '16 at 06:24
@vivman - If I've managed to answer your question please accpet it. – Yaron Oct 30 '16 at 06:24

Spark Map transformation

1 Answers1