Spark Frameless withColumnRenamed nested field

Question

Let's say I have the following code

case class MyTypeInt(a: String, b: MyType2)
case class MyType2(v: Int)
case class MyTypeLong(a: String, b: MyType3)
case class MyType3(v: Long)

val typedDataset = TypedDataset.create(Seq(MyTypeInt("v", MyType2(1))))
typedDataset.withColumnRenamed(???, typedDataset.colMany('b, 'v).cast[Long]).as[MyTypeLong]

How can I implement this transformation when the field that I am trying to transform is nested? the signature of withColumnRenamed asks for a Symbol in the first parameter so I don't know how to do this...

I am just trying to cast the column `b.v` from `Int` to `Long`, but the question is about how to transform a column that is nested like `b.v` — Mikel San Vicente, Nov 26 '18 at 19:35

score -1 · Answer 1 · answered Nov 27 '18 at 05:39

withColumnRenamed does not allow you to transform a column. To do that, you should use withColumn. One approach would then be to cast the column and recreate the struct.

scala> val new_ds = ds.withColumn("b", struct($"b.v" cast "long" as "v")).as[MyTypeLong]
scala> new_ds.printSchema
root
|-- a: string (nullable = true)
|-- b: struct (nullable = false)
|    |-- v: long (nullable = true)

Another approach would be to use map and build the object yourself:

ds.map{ case MyTypeInt(a, MyType2(b)) => MyTypeLong(a, MyType3(b)) }

I was asking to solve this with frameless API, not DataFrame API, thanks anyway — Mikel San Vicente, Nov 27 '18 at 08:54

Spark Frameless withColumnRenamed nested field

1 Answers1