Spark: How to convert RDD with many columns into a Data Frame

Question

I've converted RDDs to Data Frame using case classes, but my current data has 700 columns. I've come across mentions of using structtypes but I can't find an example. Hope someone can share an example here. Thank you. Kevin

Answer 1

Here is an example with sample input using structType :

a,1,2.0

b,2,3.0

import org.apache.spark.sql.Row
import org.apache.spark.sql.SQLContext
import org.apache.spark.sql.types.StructType
import org.apache.spark.sql.types.StructField
import org.apache.spark.sql.types.StringType
import org.apache.spark.sql.types.IntegerType
import org.apache.spark.sql.types.DoubleType

def getSchema(): StructType = {
val schema = new StructType(Array(
  StructField("col_a", StringType, nullable = true),
  StructField("col_b", IntegerType, nullable = true),
  StructField("col_c", DoubleType, nullable = true)
))
schema
}

val sqlContext = new org.apache.spark.sql.SQLContext(sc)
val rdd = sc.textFile("/tmp/test").map(m => m.split(",", -1)).map(m => Row(m(0),m(1).toInt,m(2).toDouble))
val df = sqlContext.createDataFrame(rdd, getSchema)
df.show
+-----+-----+-----+
|col_a|col_b|col_c|
+-----+-----+-----+
|    a|    1|  2.0|
|    b|    2|  3.0|
+-----+-----+-----+