convert string into date in spark dataframe using spark sql

Question

i'm very new to spark world . im using spark version 2.3.1. i try to convert string into date format in dataframe i.e "DD-MM-YYYY", however i googled and tried using cast,to_date function its not working . i don't know why .

this is my schema :

 b.printSchema
root
 |-- Emp: string (nullable = true)
 |-- Doj: string (nullable = true)

My data in dataframe:

b.select($"Emp",$"Doj").show()
+---+----------+
|Emp|       Doj|
+---+----------+
|  a|20-08-2013|
|  b|24-05-1945|
|  c|13-07-2007|
+---+----------+

what i tried

b.select($"Emp",$"Doj".cast("date")).show()
+---+----+
|Emp| Doj|
+---+----+
|  a|null|
|  b|null|
|  c|null|
+---+----+



b.select($"Emp",to_date($"Doj","format")).show()
+---+------------------------+
|Emp|to_date(`Doj`, 'format')|
+---+------------------------+
|  a|                    null|
|  b|                    null|
|  c|                    null|
+---+------------------------+

why im getting only nulls???

Does this answer your question? [Converting date from string to date in sparksql](https://stackoverflow.com/questions/52224651/converting-date-from-string-to-date-in-sparksql) — Aaron_ab, Feb 10 '21 at 10:12

score 1 · Accepted Answer · answered Aug 15 '18 at 13:41

why im getting only nulls???

Here

b.select($"Emp",$"Doj".cast("date")).show()

because data is not in a ISO 8601 compatible format.

Here

b.select($"Emp",to_date($"Doj","format")).show()

because "format" is well, not a format.

Use

b.select($"Emp",to_date($"Doj", "dd-MM-yyyy")).show()

convert string into date in spark dataframe using spark sql

1 Answers1