0

I have a string with various lengths, for example 11/8/16 and 1/27/16. The format is Month/Day/Year, how can I convert these to dates? I tried various combinations of MM, mm, DD etc but cant get it to work.

MLEN
  • 2,162
  • 2
  • 20
  • 36
  • Does this answer your question? [Convert pyspark string to date format](https://stackoverflow.com/questions/38080748/convert-pyspark-string-to-date-format) – Sharma Jan 16 '23 at 17:20

1 Answers1

0

I checked that on Spark 3, had to change timeParserPolicy to LEGACY

import pyspark.sql.functions as F

spark.conf.set("spark.sql.legacy.timeParserPolicy","LEGACY")

data1 = [
    ["11/8/16"],
    ["1/27/16"]
]

df = spark.createDataFrame(data1).toDF("source")

tmp = df.withColumn("parsed_to_date", F.to_date(F.col("source"), "MM/dd/yy"))
tmp.show(truncate = False)

output

+-------+--------------+
|source |parsed_to_date|
+-------+--------------+
|11/8/16|2016-11-08    |
|1/27/16|2016-01-27    |
+-------+--------------+
M_S
  • 2,863
  • 2
  • 2
  • 17