1

For my learning , i have been using below sample dataset .

+-------------------+-----+-----+-----+-----+-------+
|             MyDate| Open| High|  Low|Close| Volume|
+-------------------+-----+-----+-----+-----+-------+
|2006-01-03 00:00:00|983.8|493.8|481.1|492.9|1537660|
|2006-01-04 00:00:00|979.6|491.0|483.5|483.8|1871020|
|2006-01-05 00:00:00|972.2|487.8|484.0|486.2|1143160|
|2006-01-06 00:00:00|977.8|489.0|482.0|486.2|1370250|
|2006-01-09 00:00:00|973.4|487.4|483.0|483.9|1680740|
+-------------------+-----+-----+-----+-----+-------+

I tried to change "MyDate" column values to different format like "YYYY-MON" and written like this..

citiDataDF.withColumn("New-Mydate",to_timestamp($"MyDate", "yyyy-MON")).show(5)

After executing the code, found that new column "New-Mydate". but i couldn't see the desired output format. can you please help

Learn Hadoop
  • 2,760
  • 8
  • 28
  • 60

1 Answers1

1

You need date_format instead to_timestamp:

val citiDataDF = List("2006-01-03 00:00:00").toDF("MyDate")
citiDataDF.withColumn("New-Mydate",date_format($"New-Mydate", "yyyy-MMM")).show(5)

Result:

+-------------------+----------+
|             MyDate|New-Mydate|
+-------------------+----------+
|2006-01-03 00:00:00|  2006-Jan|
+-------------------+----------+

Note: Three "M" mean the month as string, if you want a month as Int, you must use only two "M"

Pablo López Gallego
  • 690
  • 4
  • 16
  • Lopez : sorry there is no change in the output. – Learn Hadoop Jun 04 '19 at 08:00
  • +-------------------+-----+-----+-----+-----+-------+-------------------+ | MyDate| Open| High| Low|Close| Volume| New-Mydate| +-------------------+-----+-----+-----+-----+-------+-------------------+ |2006-01-03 00:00:00|490.0|493.8|481.1|492.9|1537660|2006-01-03 00:00:00| – Learn Hadoop Jun 04 '19 at 08:00
  • 1
    Lopez.. Great .. thanks ..it is working as expected – Learn Hadoop Jun 04 '19 at 08:53