converting dd-MMM-yy date format in Spark

Question

I have a Spark data frame (Scala API) which contains a column called transfer date, the dates are in string format and are in this format 24-JUL-17.

I want to convert it to date string into timestamp. How can I do it?

score 5 · Answer 1 · answered May 10 '18 at 09:35

5

It can also be done with to_timestamp(date_col, expr)

import org.apache.spark.sql.functions.to_timestamp

val df = date.withColumn("ts", to_timestamp($"transfer date", "dd-MMM-yy"))

Now ts column in df is of timestamp type.

answered May 10 '18 at 09:35

I have date column in my dataframe like ...01-01-2012 , 31-01-2012 , how to convert like 01-Jan-2012 , 31-Jan-2012 format ? How to filter the data between these two date??? – BdEngineer Dec 12 '18 at 09:48

score 4 · Answer 2 · edited Jul 24 '17 at 16:54

4

I found it :

import org.apache.spark.sql.functions.unix_timestamp

val ts = unix_timestamp($"transfer date", "dd-MMM-yy").cast("timestamp")
dfs.withColumn("ts",ts).show()

edited Jul 24 '17 at 16:54

Ramesh Maharjan

41,071
6
69
97

answered Jul 24 '17 at 15:43

OUMOUSS_ELMEHDI

499
5
16

https://stackoverflow.com/questions/40763796/convert-date-from-string-to-date-format-in-dataframes/55465096#55465096 – Minkymorgan Apr 02 '19 at 21:21

converting dd-MMM-yy date format in Spark

2 Answers2