Spark: Split is not a member of org.apache.spark.sql.Row

Question

Below is my code from Spark 1.6. I am trying to convert it to Spark 2.3 but I am getting error for using split.

Spark 1.6 code:

val file = spark.textFile(args(0))
val mapping = file.map(_.split('/t')).map(a => a(1))
mapping.saveAsTextFile(args(1))

Spark 2.3 code:

val file = spark.read.text(args(0))
val mapping = file.map(_.split('/t')).map(a => a(1)) //Getting Error Here
mapping.write.text(args(1))

Error Message:

value split is not a member of org.apache.spark.sql.Row

score 5 · Accepted Answer · answered Aug 04 '19 at 14:07

5

Unlike spark.textFile which returns a RDD, spark.read.text returns a DataFrame which is essentially a RDD[Row]. You could perform map with a partial function as shown in the following example:

// /path/to/textfile:
// a    b   c
// d    e   f

import org.apache.spark.sql.Row

val df = spark.read.text("/path/to/textfile")

df.map{ case Row(s: String) => s.split("\\t") }.map(_(1)).show
// +-----+
// |value|
// +-----+
// |    b|
// |    e|
// +-----+

answered Aug 04 '19 at 14:07

Leo C

22,006
3
26
39

4

You can also `df.as[String].map(line => line.split(...)).show()`. – Luis Miguel Mejía Suárez Aug 04 '19 at 14:09
df.as[String] only works if the strings are small. Although there is a limit that can be increased this does not work well with long strings.... – Juan Zamora May 27 '20 at 18:11

Spark: Split is not a member of org.apache.spark.sql.Row

1 Answers1