-3

I have a String in the format of "MMM-YY" (ie) "jun-22","Jan-22" etc. I want to convert it into Date with 01st Day of Month in the following format.

Jan-22 --> 01-Jan-22
Feb-21 --> 01-Feb-21

I have tried a few ways but couldn't get to the solution. Can someone please advise on what is the quickest and most efficient way of doing this in a Pyspark Dataframe.

Code used could be pyspark or Python.

Pysparker
  • 93
  • 11
  • 1
    lowercase `y` for year in pyspark -- see [available patterns](https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html). also share what you've tried – samkart Nov 07 '22 at 08:04
  • Does this answer your question? [Convert pyspark string to date format](https://stackoverflow.com/questions/38080748/convert-pyspark-string-to-date-format) – samkart Nov 07 '22 at 08:07
  • I couldn't figure out how to do that, so just tried converting using from_unixtime, but that didn't work – Pysparker Nov 07 '22 at 08:22
  • 3
    If it's a string, why can't you just prepend '01-' ? – DarkKnight Nov 07 '22 at 08:24
  • If stuck, this may get you out of the hook `spark.sql("set spark.sql.legacy.timeParserPolicy=LEGACY") df1 = sqlContext.createDataFrame([("Jan-22","11/24/1991","11/30/1991"), ("Feb-21","11/24/1992","11/30/1992")], schema=['first', 'second', 'third']) df1.show() new = (df1.withColumn('first',split(col('first'),'-'))#Split string by '-' to create list .withColumn('first',concat_ws('-',lit('01'),*[col("first")[i] for i in range(2)]))#create string date .withColumn('date_desired',F.expr(" to_date(first ,'dd-MMMM-yy') "))#coerce to date )` – wwnde Nov 07 '22 at 10:59
  • Hi, you should include examples of what you've already tried and how didn't it work. – Gustavo Puma Nov 08 '22 at 16:25

1 Answers1

0

Thanks for the help. I was able to add "01-" at the beginning of the date string and converting it into a date.

MAhmad
  • 5
  • 5