How to set column names to toDF() function in spark dataframe using a string array?

Question

For example,

val columns=Array("column1", "column2", "column3")
val df=sc.parallelize(Seq(
(1,"example1", Seq(0,2,5)),
(2,"example2", Seq(1,20,5)))).toDF(columns)

How can I set column name using string Array? Is it possible to mention data types inside toDF()?

score 14 · Accepted Answer · answered Jun 23 '16 at 13:16

14

toDF() takes a repeated parameter of type String, so you can use the _* type annotation to pass a sequence:

val df=sc.parallelize(Seq(
  (1,"example1", Seq(0,2,5)),
  (2,"example2", Seq(1,20,5)))).toDF(columns: _*)

For more on repeated parameters - see section 4.6.2 in the Scala Language Specification.

answered Jun 23 '16 at 13:16

Tzach Zohar

37,442
3
79
85

2

Thanks..How to specify data type along with columns inside toDF()? – Devi Jun 24 '16 at 04:25
Thanksss.... this is exactly i'm searching for morre than an hour,,, thanks – Vicky Aug 18 '17 at 23:37

score 8 · Answer 2 · answered Jun 23 '16 at 13:08

8

val df=sc.parallelize(Seq(
(1,"example1", Seq(0,2,5)),
(2,"example2", Seq(1,20,5)))).toDF("column1", "column2", "column3")

toDF() takes comma-seperated strings

answered Jun 23 '16 at 13:08

anshul_cached

684
5
18

score 6 · Answer 3 · edited May 23 '17 at 11:33

6

toDF() is defined in Spark documentation as:

def toDF(colNames: String*): DataFrame

And so you need to turn your array to a varargs as also described here. That means you need to do the following:

val columns=Array("column1", "column2", "column3")
val df=sc.parallelize(Seq(
(1,"example1", Seq(0,2,5)),
(2,"example2", Seq(1,20,5)))).toDF(columns: _*)

(Add : _* tocolumns in toDF)

edited May 23 '17 at 11:33

Community

1
1

answered Jun 23 '16 at 13:24

shakedzy

2,853
5
32
62

How to set column names to toDF() function in spark dataframe using a string array?

3 Answers3