How to transform dataset to another

Question

My dataset has columns named "key(string), value(long)"

The value of column key like prefix.20171012.111.2222, and the value of column value like 9999.

I want to transform the dataset to a new one which split the colmun key to others like ths "day, rt, item_id, value".

how to do it, thanks a lot

Maybe this question can help you: https://stackoverflow.com/questions/39255973 — Shaido, Oct 12 '17 at 03:46

score 0 · Accepted Answer · answered Oct 12 '17 at 06:54

// input ds looks like this
+--------+-----+
|     key|value|
+--------+-----+
|20171011| 9999|
+--------+-----+

//import the functions you need
import org.apache.spark.sql.functions.{to_date, month, year, dayofmonth}

// ds2 
val ds2 = ds.withColumn("date", to_date($"key", "yyyyMMdd"))

// ds2.show()
+--------+-----+----------+
|     key|value|      date|
+--------+-----+----------+
|20171011| 9999|2017-10-11|
+--------+-----+----------+

// ds3
val ds3 = ds2.withColumn("Month", month($"date"))
  .withColumn("Year", year($"date"))
  .withColumn("Date", dayofmonth($"date"))

// ds3.show()
+--------+-----+----+-----+----+
|     key|value|Date|Month|Year|
+--------+-----+----+-----+----+
|20171011| 9999|  11|   10|2017|
+--------+-----+----+-----+----+

How to transform dataset to another

1 Answers1