dataframe too many arguments in the rdd object

Question

I tryed to use this question to convert rdd object to dataframe in spark. The class in my use case contains more than 100 arguments (columns)

case class MyClass(val1: String, ..., val104: String )

    val df = rdd.map({ 
      case Row(val1: String, ..., val104: String) => MyClass(val1, ..., val104)
    }).toDF("col1_name", ..., "col104_name")

I got this Error: too many arguments for unapply pattern, maximum = 22

May someone help me with a concrete example; i'm using spark 1.6 with scala . Thank you

you can refer this question http://stackoverflow.com/questions/41799147/defining-dataframe-schema-for-a-table-with-1500-columns-in-spark/41799237#41799237 — Rajat Mishra, Apr 03 '17 at 12:21
I read that question but they require a csv file but i'm not using it; i'm using xl3 file — Zied Hermi, Apr 03 '17 at 13:25
the problem with case class is that it has limitation of 22 fields in scala version < 2.11 — Rajat Mishra, Apr 03 '17 at 13:29

score 2 · Answer 1 · answered Apr 03 '17 at 11:24

Your problem is the limitation on case classes to 22 fields.

You need to define your case class as a more structured data type, so that there are only 22 top fields (but some of those may be case classes again).

Once you've done this, you can use your Row (which itself cannot be pattern matched with 104 fields, you will have to use row(0), row(1), ..., row(103)) to build your MyClass.

dataframe too many arguments in the rdd object

1 Answers1